lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 19.8613 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.2565 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.2609 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/pipeline_orig.qgraph DEEP/20220821/B1d/science#step1/20240307T003716Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.3043 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 14.9276 seconds; current memory usage: 0.368 Gibyte, delta: 0.190 Gibyte, peak delta: 0.178 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.368 Gibyte (main), 0.360 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0214 seconds; current memory usage: 0.368 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.368 Gibyte (main), 0.360 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 744 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220821_B1d_science#step1_20240307T003716Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.3724 seconds; current memory usage: 0.369 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.369 Gibyte (main), 0.360 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 746 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.9880 seconds; current memory usage: 0.370 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.370 Gibyte (main), 0.360 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: af6ddbd0-769f-4e1a-a147-8dbbbadd5895 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 00:38:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 00:38:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 00:38:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 2024-03-07 00:38:27 proc_lsst.multi:170 [INFO] [multi] job_id 44080 proc_lsst.multi INFO: [multi] job_id 44080 2024-03-07 00:38:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:27 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 44080 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 44080 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 44080 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 043156df027a00d1102ad3a2342a4dfa parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3e6d56cf-2b80-42f8-988d-9a9b9f64002c_isr_1124086_16.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3e6d56cf-2b80-42f8-988d-9a9b9f64002c_isr_1124086_16.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 11f94c603985f5af09514a298fa85076 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/581163fa-c8a1-4f4e-8a79-ee3577a9c577_isr_1124086_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/581163fa-c8a1-4f4e-8a79-ee3577a9c577_isr_1124086_58.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 20dd0c24ed196a390b56894981c1d229 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/501f8994-e8b0-451d-91e9-8d9771a12f0b_isr_1123955_43.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/501f8994-e8b0-451d-91e9-8d9771a12f0b_isr_1123955_43.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 1d922f9eeea1423f9e0d3849a798db74 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63906200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8627d4ed-f210-425a-a4cf-bb7bfeb16d2a_isr_1123953_36.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8627d4ed-f210-425a-a4cf-bb7bfeb16d2a_isr_1123953_36.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 7eb85f4462636768740cb6a883584dd2 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/97ab3281-73ab-41c8-b11a-d92d62c9a1bb_isr_1123953_3.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/97ab3281-73ab-41c8-b11a-d92d62c9a1bb_isr_1123953_3.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 53a492538a905c2df27ae4202c0efe53 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63906a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b224c961-6136-4a8b-ba17-4b70bbd03b9d_isr_1123957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b224c961-6136-4a8b-ba17-4b70bbd03b9d_isr_1123957_53.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 24a7edf66c3f8c3e06d64cf8017c21f6 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63906de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/885dae52-e8e6-4047-9eee-82b355f71dc7_isr_1123955_53.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/885dae52-e8e6-4047-9eee-82b355f71dc7_isr_1123955_53.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 1adbdfe46a251855d16183d7fbac1716 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2fedd920-f4cc-45e1-9e1f-e5ddcef86912_isr_1124086_51.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2fedd920-f4cc-45e1-9e1f-e5ddcef86912_isr_1124086_51.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 35d19d2bde2222e71dc6f7fcda17eb2e parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7356a218-b4e5-40f0-8332-0688c6f1608c_isr_1123955_10.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7356a218-b4e5-40f0-8332-0688c6f1608c_isr_1123955_10.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 2be62b5ea9a5939135fac3b8b758e71a parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/227c0efa-509a-42ad-baad-60f92f25cadc_isr_1123957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/227c0efa-509a-42ad-baad-60f92f25cadc_isr_1123957_29.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 5825613a12f6e91ea9d85ee88114e344 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9a293480-7701-4dd5-bd8b-79026dc2d983_isr_1124086_19.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9a293480-7701-4dd5-bd8b-79026dc2d983_isr_1124086_19.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 62c6f3bba935aab64f27166365d25333 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ad05ab9f-ad6c-493a-9145-5c4c54a3a65a_isr_1123953_13.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ad05ab9f-ad6c-493a-9145-5c4c54a3a65a_isr_1123953_13.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 9102fc7bcedce4393f646f790be3cb75 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a8e6cede-ebfe-4cb6-8c4a-86b7ec508a91_isr_1123953_57.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a8e6cede-ebfe-4cb6-8c4a-86b7ec508a91_isr_1123953_57.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 93482d01f8607980fde192fefc201933 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f0d9df33-2f3e-46b0-967d-d054045d72b9_isr_1123955_59.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f0d9df33-2f3e-46b0-967d-d054045d72b9_isr_1123955_59.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash f37c12fa5b064913c5989bab33becc20 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/bb83d0d6-8e21-4663-8f0e-d336b97c5d32_isr_1123953_8.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/bb83d0d6-8e21-4663-8f0e-d336b97c5d32_isr_1123953_8.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash cd2d4db0e9d8bd841d5776a109eb4adb parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad6ec155-d552-4999-8fdd-55922c978386_isr_1123955_37.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad6ec155-d552-4999-8fdd-55922c978386_isr_1123955_37.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash ab35e444ff58c1e029a9a9f557b13141 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/509fcdcf-5b31-4f5c-b1b5-54ee779715b0_isr_1123953_40.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/509fcdcf-5b31-4f5c-b1b5-54ee779715b0_isr_1123953_40.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 95fdd8dd4e0f2502c9fd3f530f3e77c0 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2f6bded2-aa6c-4fab-96ff-78c0be943b47_isr_1124086_62.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2f6bded2-aa6c-4fab-96ff-78c0be943b47_isr_1124086_62.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 8e8e1294ee3dabe05ca41e7163541e72 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea4f46f1-85a5-4476-984d-3d4e76eb4cc1_isr_1123953_42.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea4f46f1-85a5-4476-984d-3d4e76eb4cc1_isr_1123953_42.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash aca7e485cfcbb14899b8e1f92bec6f36 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef3e10c1-687c-41e8-92b3-7cb24360a933_isr_1123955_32.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef3e10c1-687c-41e8-92b3-7cb24360a933_isr_1123955_32.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 7a48c636e87828e7dbea4b9728e1612a parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/70e27f0d-9e33-4f87-bdaf-eff45e212c5a_isr_1123955_11.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/70e27f0d-9e33-4f87-bdaf-eff45e212c5a_isr_1123955_11.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 3bd1ec8bbeb2fd14b139a1cf99160ff0 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f25c9a1d-d09f-4d55-925e-a07738c6cdcf_isr_1123955_9.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f25c9a1d-d09f-4d55-925e-a07738c6cdcf_isr_1123955_9.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 0b67b550fc0bc20c717a5bbfcf99ec7c parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e568a80e-8c97-4532-89b6-9ee520a406ef_isr_1123957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e568a80e-8c97-4532-89b6-9ee520a406ef_isr_1123957_54.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 7d380bd4c596f0d5bf18477e2f3f1ae1 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/545b81a6-dbc0-4319-bb73-e7c542fd87de_isr_1123955_5.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/545b81a6-dbc0-4319-bb73-e7c542fd87de_isr_1123955_5.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash a6200c58dd5525978a5028e2588273cd parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bbd531ba-d412-49c3-a9f3-27b34e690a79_isr_1124086_47.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bbd531ba-d412-49c3-a9f3-27b34e690a79_isr_1124086_47.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 11f127693f59f5f840fdebdb3fc0d7cb parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/2464b019-443b-40a7-b470-f42bd9e4eb8a_isr_1123953_53.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/2464b019-443b-40a7-b470-f42bd9e4eb8a_isr_1123953_53.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash fff401ab8e83dfd4c7b54d69f01343f5 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/afe1ab77-7f0d-4019-b96c-6add22d747d8_isr_1124086_44.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/afe1ab77-7f0d-4019-b96c-6add22d747d8_isr_1124086_44.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 96e71c4dffa07de08ffca5b108fc295e parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6db83033-6bb0-416c-8677-93514b0c4568_isr_1123955_41.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6db83033-6bb0-416c-8677-93514b0c4568_isr_1123955_41.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 4ca692df6613658ba6032f9bffab43be parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cbf966c-2fea-4bd4-9548-c4ab8a6f0a21_isr_1123955_13.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cbf966c-2fea-4bd4-9548-c4ab8a6f0a21_isr_1123955_13.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 2ab6b36c4b8c2dd9371627c433eb27ec parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/722e649f-7d53-4544-add4-d2e37d4ddebb_isr_1123955_47.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/722e649f-7d53-4544-add4-d2e37d4ddebb_isr_1123955_47.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 571e4a9c433883ed5d3e37fe02938789 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e5dab166-5351-4acc-84d3-da68ed102290_isr_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e5dab166-5351-4acc-84d3-da68ed102290_isr_1123953_31.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 2f0f2d8a60700913d602e4023876321b parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/890cb3a2-eea6-40e9-9c0c-31271d7851c2_isr_1123955_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/890cb3a2-eea6-40e9-9c0c-31271d7851c2_isr_1123955_23.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 98bfa41223aafa2715fdfade2f963fa8 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/ef56186b-3a83-4004-9477-722d911df0d4_isr_1124086_48.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/ef56186b-3a83-4004-9477-722d911df0d4_isr_1124086_48.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash d816b163ed57e7b03fe4c47c4a74dc59 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/de5dfd98-6ba8-4406-91b8-cc18df0150cb_isr_1124086_9.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/de5dfd98-6ba8-4406-91b8-cc18df0150cb_isr_1124086_9.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 51d8e56b1c05a061c3ccc219bedb8823 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5619e658-549b-4360-89cd-1ed90a2e5623_isr_1123957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5619e658-549b-4360-89cd-1ed90a2e5623_isr_1123957_19.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash e7c38ed3466399ffa158dca4c2127291 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f2f82aa1-8e5d-431d-94c8-77d22cf49018_isr_1123957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f2f82aa1-8e5d-431d-94c8-77d22cf49018_isr_1123957_7.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 74807a5a0a02a33c34038b5b387b750c parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f7b0ac41-5758-44bc-8d26-133b6cb3bb28_isr_1124086_57.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f7b0ac41-5758-44bc-8d26-133b6cb3bb28_isr_1124086_57.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 3b76182796ca4c2027c3da41dd4dbd33 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/321d21a6-3ed7-46b0-b2c3-e85195bfebc9_isr_1123957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/321d21a6-3ed7-46b0-b2c3-e85195bfebc9_isr_1123957_45.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash a79105beea3bfdb6175e0303b9ca7564 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f6f66d12-7162-4fb2-b611-49f235ad6b94_isr_1123953_30.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f6f66d12-7162-4fb2-b611-49f235ad6b94_isr_1123953_30.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 4bb4953a96c90856b9b7edac48d76326 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70ccac7a-126c-4146-8cc5-124fc02d3727_isr_1124086_49.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70ccac7a-126c-4146-8cc5-124fc02d3727_isr_1124086_49.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 0fbd2d27ce811221fdbe5219c9599661 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/342b720e-3ee4-470e-ab2a-dd2e2fd5a0fc_isr_1123957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/342b720e-3ee4-470e-ab2a-dd2e2fd5a0fc_isr_1123957_58.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 48ab3e8635073de353fa0c2fc17d41f0 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/af3ba71f-b1ae-451a-aef8-4b9bbfb96880_isr_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/af3ba71f-b1ae-451a-aef8-4b9bbfb96880_isr_1123955_19.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash ac714661946f13bb2ba0b936076888d5 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d6284207-b9b4-479e-a137-50cf03bc4c5c_isr_1123955_39.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d6284207-b9b4-479e-a137-50cf03bc4c5c_isr_1123955_39.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 69c7efa4aba4b47f4617a1cb418e5693 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/15939763-0fda-4c71-83bf-fc207666c5f6_isr_1123953_37.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/15939763-0fda-4c71-83bf-fc207666c5f6_isr_1123953_37.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash d21b770ad5ecff85ff6da081ed397ca0 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/999ec973-32f1-4024-a44a-2127c724a03e_isr_1123953_62.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/999ec973-32f1-4024-a44a-2127c724a03e_isr_1123953_62.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash abcddd369c6b5e6da311432153290124 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e6d89b66-6a8b-476c-8264-876f03ec05fb_isr_1123957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e6d89b66-6a8b-476c-8264-876f03ec05fb_isr_1123957_3.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 1e38624d56e0208f78b5ffceffc2a930 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/918c0595-5c5a-40f1-ace8-3e0ad51201e9_isr_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/918c0595-5c5a-40f1-ace8-3e0ad51201e9_isr_1123957_61.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash f30c801b33d7828722cd6db9439948e2 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d3528af1-da1a-4be0-9b1c-da1be68f8265_isr_1123953_44.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d3528af1-da1a-4be0-9b1c-da1be68f8265_isr_1123953_44.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 67e4cb015024ac28589276f1e12e1327 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/4d21185a-ac7e-44b1-a9e8-3c4607ddc406_isr_1123953_9.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/4d21185a-ac7e-44b1-a9e8-3c4607ddc406_isr_1123953_9.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 7213e4b9ad5998da4ee9268c0c491f08 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/0e13b284-1e2b-4560-8206-279062ea06a1_isr_1123955_48.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/0e13b284-1e2b-4560-8206-279062ea06a1_isr_1123955_48.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 286a97a9cc6a09ee4cba62a0f18cc94e parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/252fc933-e65c-47d7-b856-a74f2bf442e5_isr_1124086_18.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/252fc933-e65c-47d7-b856-a74f2bf442e5_isr_1124086_18.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 3e00a20d9373328b6147c34bdc978daa parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/84cb695b-5891-4ce4-ae16-70c9cc511a72_isr_1123955_25.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/84cb695b-5891-4ce4-ae16-70c9cc511a72_isr_1123955_25.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 7d47a0c40f2f04bb575a319322bde138 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c4bfa857-0b49-4139-a900-c4cb66f6c7ba_isr_1123953_24.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c4bfa857-0b49-4139-a900-c4cb66f6c7ba_isr_1123953_24.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 4853f7a7321bfd105cfec27971ade546 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1d7a51a4-eb7f-4d10-9282-a0c87bcb7133_isr_1123957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1d7a51a4-eb7f-4d10-9282-a0c87bcb7133_isr_1123957_10.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash aee641025521512a4761fb055325dd37 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/12a01a31-a1bc-4cc2-baf8-e9ebc76ca3ac_isr_1123957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/12a01a31-a1bc-4cc2-baf8-e9ebc76ca3ac_isr_1123957_52.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 117a6bc7ba172173dffe815bcd02a27a parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4159ac2f-f295-4751-bb15-72aaef87373c_isr_1123955_22.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4159ac2f-f295-4751-bb15-72aaef87373c_isr_1123955_22.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash e8dbf1f666c1e06f764aee903f74ca5c parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/758a6ace-ff2d-4ee3-bd29-c07fda6e6aae_isr_1124086_54.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/758a6ace-ff2d-4ee3-bd29-c07fda6e6aae_isr_1124086_54.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 2fc137a4648ff23790b22a35bdc6ad07 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/28a9ad41-2518-4a2c-b2a7-2e498c16d883_isr_1123957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/28a9ad41-2518-4a2c-b2a7-2e498c16d883_isr_1123957_16.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 4aafad119d021ebb7cd30c4aa5c5a273 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639611c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2b534a1f-3d71-494c-b225-17a8429a9f87_isr_1123955_17.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2b534a1f-3d71-494c-b225-17a8429a9f87_isr_1123955_17.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash ee0ce2ddc95a4054ea5b559024fe83bc parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639614e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/add313a8-e85b-46a8-ac3f-cb453368a488_isr_1123953_28.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/add313a8-e85b-46a8-ac3f-cb453368a488_isr_1123953_28.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash d48595587f5ddfd94e276fd2c94e31b1 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/45925fd8-d0f5-44c9-9d6f-c1038e0d339e_isr_1123955_40.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/45925fd8-d0f5-44c9-9d6f-c1038e0d339e_isr_1123955_40.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 2fe5f5302f5a9cc32889cb934a5c6b60 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5a59b231-f9d6-4ccc-b5ee-675f9a44ab82_isr_1124086_8.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5a59b231-f9d6-4ccc-b5ee-675f9a44ab82_isr_1124086_8.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 5372fc96c462c6ad92c44d4ff2db20ab parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cc6c25dc-c4f6-413d-b6c7-582f0133dfe0_isr_1123957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cc6c25dc-c4f6-413d-b6c7-582f0133dfe0_isr_1123957_50.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 4cd50f31429c330cfb8b0cdbf743c707 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e2cfd06f-a720-49c2-9a82-abb3cab17bbc_isr_1123957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e2cfd06f-a720-49c2-9a82-abb3cab17bbc_isr_1123957_22.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 2ad2813ebdadc22fd55aa6c38c97dabe parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/43f8c892-8357-4e3e-ab2e-73b5a015101e_isr_1124086_27.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/43f8c892-8357-4e3e-ab2e-73b5a015101e_isr_1124086_27.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 35b2a987fe1fd25c78adfd200dc876ce parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639627a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6f48242d-33b2-4cd9-9263-b21e821aab3e_isr_1123957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6f48242d-33b2-4cd9-9263-b21e821aab3e_isr_1123957_33.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 9d29c057d2d5be8fa49f8b7982905d8b parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/50ac643d-587f-4cdb-b454-d7033ecaa338_isr_1123955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/50ac643d-587f-4cdb-b454-d7033ecaa338_isr_1123955_2.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 098bf7a1b42f9fb4c4c3b6701f88b6c7 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/8ae84bfe-9ece-44ee-ae34-0bf38934b29f_isr_1123957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/8ae84bfe-9ece-44ee-ae34-0bf38934b29f_isr_1123957_4.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash b8adf3773e7fa3c5bfbfc39b8f94d456 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ebdaf148-101c-45d1-b42a-b04f733c3377_isr_1123957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ebdaf148-101c-45d1-b42a-b04f733c3377_isr_1123957_48.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 61545ad3734b025b3f486a044566af65 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c45bbd12-7e27-46dc-9b8e-494cf4af432b_isr_1123953_34.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c45bbd12-7e27-46dc-9b8e-494cf4af432b_isr_1123953_34.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 233952ebae6855f849d44208fe9f5206 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ce8b27c3-9e29-45be-a9e1-32da9819cc1d_isr_1123953_29.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ce8b27c3-9e29-45be-a9e1-32da9819cc1d_isr_1123953_29.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 5eba0a6bf6059200ef1e0f340353b3d3 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bc60964e-e2c5-4a03-a40a-7402b4eb21fa_isr_1124086_53.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bc60964e-e2c5-4a03-a40a-7402b4eb21fa_isr_1124086_53.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash cdc6d5f858cc68d16a5ee60e80d0ed9e parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/81ee7c42-1484-4a85-b66e-16dbfac5f602_isr_1123957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/81ee7c42-1484-4a85-b66e-16dbfac5f602_isr_1123957_13.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash e60f4ee58a4a6c4a3d2edd8bc2fde081 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639880e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/411c5f6b-16c9-49f2-a24d-9ac0f502babb_isr_1123955_52.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/411c5f6b-16c9-49f2-a24d-9ac0f502babb_isr_1123955_52.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 46790ed57de18621f1579ca4b63bad17 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/96905294-674e-4ab8-b117-fcf4df557c83_isr_1123955_18.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/96905294-674e-4ab8-b117-fcf4df557c83_isr_1123955_18.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash b8595f1867b9a9a9d767d73184e84b6b parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/0df06c2a-7a4f-4f31-942e-820971a360e8_isr_1123957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/0df06c2a-7a4f-4f31-942e-820971a360e8_isr_1123957_15.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash a26a6afac0efa3460e9d43ae6e920d4d parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1fdf551e-60d9-4576-bcd5-d34fa9363ec7_isr_1123957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1fdf551e-60d9-4576-bcd5-d34fa9363ec7_isr_1123957_41.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash a023b2b8d4477b45e828e093993f46b5 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/87e574c6-8b1b-4714-a6c1-67f1120d79c6_isr_1124086_12.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/87e574c6-8b1b-4714-a6c1-67f1120d79c6_isr_1124086_12.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 4a5b935273362097175e005c335f4152 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b6930d35-0bda-4021-8054-f2522b156157_isr_1124086_6.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b6930d35-0bda-4021-8054-f2522b156157_isr_1124086_6.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash ea746956fd65b4837279ddfd094fa54b parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639893a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38dba4c1-05db-41c1-97c2-740e7d4b6c77_isr_1123957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38dba4c1-05db-41c1-97c2-740e7d4b6c77_isr_1123957_40.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash d29585686f1ed6c1a8b5b416c7bee095 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639896c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a083fa1d-072e-4031-9cf5-2ed0a250a0e4_isr_1123955_6.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a083fa1d-072e-4031-9cf5-2ed0a250a0e4_isr_1123955_6.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash a16bd79d2b09e0c28651395dfd4961d0 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639899e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9d5a2370-c560-4af4-9291-b53d22b4cae3_isr_1123957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9d5a2370-c560-4af4-9291-b53d22b4cae3_isr_1123957_47.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 36aec3cb16b72f2eae497026a02a47af parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/11d5d40b-ede6-4162-a4e2-3839ef1e7798_isr_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/11d5d40b-ede6-4162-a4e2-3839ef1e7798_isr_1123955_55.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash f3edd5a6aca1e92354d3f76003c1599d parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4a280587-21b3-4279-917f-3baf566926e1_isr_1123957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4a280587-21b3-4279-917f-3baf566926e1_isr_1123957_25.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 84a11c98041aa53739624f0780f379b1 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/859c40dd-872c-4e72-a1a0-6d9f66058f2d_isr_1124086_20.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/859c40dd-872c-4e72-a1a0-6d9f66058f2d_isr_1124086_20.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 717b8e8d21a88feb83d2b9e53e72975b parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e0b19ec2-e1f3-4e93-9f22-ef0215973612_isr_1123953_1.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e0b19ec2-e1f3-4e93-9f22-ef0215973612_isr_1123953_1.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash f1e6846f056e92e1fe1b9c8934ba0c90 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d2d98d39-5c7d-4833-889c-86b94daa291c_isr_1123955_60.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d2d98d39-5c7d-4833-889c-86b94daa291c_isr_1123955_60.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 8e0c4e2c1e3a73c2025e9cc196c0fa88 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/fac09580-f172-4d94-af8d-bfab634c3611_isr_1123955_27.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/fac09580-f172-4d94-af8d-bfab634c3611_isr_1123955_27.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 06324bf5c4cf133c0484cd4d79e19bba parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0a194842-94c5-4b60-a67d-90d15c4abff2_isr_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0a194842-94c5-4b60-a67d-90d15c4abff2_isr_1123953_61.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash bbd5be11e100730fa8515f263cf7808c parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/55adb9dd-15d6-4836-a085-350ec9cbf937_isr_1123953_6.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/55adb9dd-15d6-4836-a085-350ec9cbf937_isr_1123953_6.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash b285857071561c66acad8b169094359a parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f12cc5f9-291a-40cc-8259-dee9b4ca1c6d_isr_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f12cc5f9-291a-40cc-8259-dee9b4ca1c6d_isr_1123957_31.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash cb3d86e5068b654a1110388db8584825 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/5d92111a-3e46-441c-84f0-83fc07dbda13_isr_1123953_46.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/5d92111a-3e46-441c-84f0-83fc07dbda13_isr_1123953_46.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 14d3a6768af597244ecc9c8a519c9fa2 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/dfeee673-d793-4dc8-8e83-7dbe46431492_isr_1124086_28.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/dfeee673-d793-4dc8-8e83-7dbe46431492_isr_1124086_28.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 733852fbf40834cb9b3a2ee9c35be767 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/607b01fd-d1d2-4843-989b-ffe9fced48dd_isr_1124086_38.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/607b01fd-d1d2-4843-989b-ffe9fced48dd_isr_1124086_38.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 9c26c2991fb8acfaf59966ba7b09716c parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/73adf95c-d633-4d7a-aa3d-bb56a851cf26_isr_1123953_4.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/73adf95c-d633-4d7a-aa3d-bb56a851cf26_isr_1123953_4.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 53b9c9b0ab6b6c4de378be6cc0163d25 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a87a2d74-4e2e-41ff-8b23-dae7a6393b11_isr_1123953_33.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a87a2d74-4e2e-41ff-8b23-dae7a6393b11_isr_1123953_33.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash c9ae8d8055e63acafc60a2e6476a3904 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c363dfa6-e6d6-48b1-b013-69191048b580_isr_1124086_5.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c363dfa6-e6d6-48b1-b013-69191048b580_isr_1124086_5.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash e901e508de63e380becd306ccd9482b3 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639acc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8ce1abe8-da2b-4362-87cf-0ae1fb6f8008_isr_1123955_36.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8ce1abe8-da2b-4362-87cf-0ae1fb6f8008_isr_1123955_36.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash a845e633d894c9ed79d28e78b51e42ae parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639acf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/90ca82b6-d60d-480e-befe-976c81cde2a7_isr_1124086_30.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/90ca82b6-d60d-480e-befe-976c81cde2a7_isr_1124086_30.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 3531d8c7125cbc747ac60700b7cd08b5 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a6ff9c76-0c4f-4d2f-aad8-4b7b60d101ab_isr_1123955_12.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a6ff9c76-0c4f-4d2f-aad8-4b7b60d101ab_isr_1123955_12.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash ab7e411644fcfc9f18b101946bd2794b parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f97e29f6-b586-4f10-841f-50f0ffe77a94_isr_1123957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f97e29f6-b586-4f10-841f-50f0ffe77a94_isr_1123957_43.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 91cd330df7866fda6852a93e6359ae86 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b8d50c64-a75a-4f4f-8f13-5182ac3538ac_isr_1123957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b8d50c64-a75a-4f4f-8f13-5182ac3538ac_isr_1123957_42.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash e6660926b02e3422f49ea6d13bd5d92d parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639adbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/47cbe57f-413a-4a65-8ad6-579c24a5301e_isr_1124086_32.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/47cbe57f-413a-4a65-8ad6-579c24a5301e_isr_1124086_32.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash bec6cc7a3ccdd1ecb73aa5c07edc6934 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639adee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/26ef2bcf-60b9-4c72-8dad-89ab38146fc8_isr_1124086_36.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/26ef2bcf-60b9-4c72-8dad-89ab38146fc8_isr_1124086_36.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 8da1d0c85b3c02f88204d556ab4feddf parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba6938ef-895a-44c9-a071-3512e5872cec_isr_1123957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba6938ef-895a-44c9-a071-3512e5872cec_isr_1123957_51.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 3d92cd47e512086bb21bb7f8637abd24 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/fb2d698b-2d10-42b5-9af7-e75a5b5c558a_isr_1123957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/fb2d698b-2d10-42b5-9af7-e75a5b5c558a_isr_1123957_62.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 99895d7d6d49eb894c0b48f5b18a1978 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7cf81efa-6133-4f52-9194-ff317b7390ca_isr_1123955_14.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7cf81efa-6133-4f52-9194-ff317b7390ca_isr_1123955_14.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash cbf8ff1b53cf76006fb87827b062d5f4 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/24d59b2b-68d0-4d08-b3d5-0f5e6a771b3c_isr_1124086_10.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/24d59b2b-68d0-4d08-b3d5-0f5e6a771b3c_isr_1124086_10.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 3ba8e7b158ea0fa2e71edb1a934481f1 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70b437a1-2f85-4b38-8f31-9d6397edf1ad_isr_1124086_34.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70b437a1-2f85-4b38-8f31-9d6397edf1ad_isr_1124086_34.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 63829d248e85ddd62157f31483c53d42 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6b00b6e9-4f9d-41fa-a50a-d0907247f3ac_isr_1123957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6b00b6e9-4f9d-41fa-a50a-d0907247f3ac_isr_1123957_20.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 29175837f0dc74f3589078eed6212a75 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/63bcc597-5b01-4854-b732-20264eb940e8_isr_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/63bcc597-5b01-4854-b732-20264eb940e8_isr_1123955_42.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 40c6e7ac23cfb904469a06518831c5ef parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5d9f7c82-8bfe-4d1b-ba68-1a2a09dfa2b5_isr_1123955_20.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5d9f7c82-8bfe-4d1b-ba68-1a2a09dfa2b5_isr_1123955_20.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 6307e08b1ea521635c13ee3aaadc260c parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639afb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/23be0b7c-9250-4197-b69e-f34362de0ef9_isr_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/23be0b7c-9250-4197-b69e-f34362de0ef9_isr_1123957_24.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 49e3a8af594917f26ac66ee0f51a5f1e parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639afe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/841f4046-60e9-46d7-8ced-d6d527cf5ce7_isr_1123955_45.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/841f4046-60e9-46d7-8ced-d6d527cf5ce7_isr_1123955_45.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash b7ac1ed4a04f38c2a572c994f5a1866f parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/666772c0-15df-4f20-8be1-2723440d7aab_isr_1124086_23.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/666772c0-15df-4f20-8be1-2723440d7aab_isr_1124086_23.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 93833ab19947a1e7aa33127677fffc83 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f623d912-ff39-450e-ab8b-a178bc400b1e_isr_1124086_50.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f623d912-ff39-450e-ab8b-a178bc400b1e_isr_1124086_50.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 9bae29a0e241327a348510ed6c349e02 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4326f2d6-a7ba-4f7e-8b43-1aa8c4bfd7d7_isr_1123955_54.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4326f2d6-a7ba-4f7e-8b43-1aa8c4bfd7d7_isr_1123955_54.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 80374959410c3302f421a99514b2d87b parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6372152b-e948-47cd-8d19-3a0e565f755c_isr_1123953_22.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6372152b-e948-47cd-8d19-3a0e565f755c_isr_1123953_22.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash a7bc8e6e911ad450a21c06bc9d0b3c5c parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/7b72d5d2-1d77-48cb-a77d-6f38ecc80577_isr_1123953_56.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/7b72d5d2-1d77-48cb-a77d-6f38ecc80577_isr_1123953_56.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 32d85266cfc7e7ca04a3a11311c39549 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/11c09840-db04-4dd7-9294-d6103cfaa535_isr_1123957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/11c09840-db04-4dd7-9294-d6103cfaa535_isr_1123957_38.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash b542812d5a891441db49d21f2efc0f2b parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/118fbaa3-3b3d-4162-a743-1d2b62a368e5_isr_1124086_43.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/118fbaa3-3b3d-4162-a743-1d2b62a368e5_isr_1124086_43.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 2fb0081648040be66b910d0f3b0ffe70 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d321e201-d648-4e7e-a0da-46a2698c2fdc_isr_1123953_47.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d321e201-d648-4e7e-a0da-46a2698c2fdc_isr_1123953_47.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 4e9c580887006680b9a37702bd261340 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/12790f74-a77b-413f-a9d3-d723bb417209_isr_1124086_15.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/12790f74-a77b-413f-a9d3-d723bb417209_isr_1124086_15.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash ae54efc87d10c935bf7cbf87fd1681b5 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/eb82beb1-20bf-420d-a3b1-d77b219f2b7f_isr_1123957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/eb82beb1-20bf-420d-a3b1-d77b219f2b7f_isr_1123957_27.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 54104cd9c42aa46e857015c90c08297e parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e24fbfde-981e-4c81-97fd-6cc8a09b4fe6_isr_1123953_51.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e24fbfde-981e-4c81-97fd-6cc8a09b4fe6_isr_1123953_51.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 772ad20c9445c77ac47e614a41e2f693 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/a23b154c-299b-44c3-81cd-d191de6055e8_isr_1124086_7.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/a23b154c-299b-44c3-81cd-d191de6055e8_isr_1124086_7.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash e2e8e82d88b0b52b2316bd62a44e29a6 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/20db20b6-a976-4252-b5e6-df5610c8998e_isr_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/20db20b6-a976-4252-b5e6-df5610c8998e_isr_1123957_28.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 13fde97b3fd8f2086ea0b2a5140f0f90 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639daa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea80c917-e965-4b79-a523-857785a75829_isr_1123953_7.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea80c917-e965-4b79-a523-857785a75829_isr_1123953_7.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 40e781a16e76e73742e26b7c2ce4a5b9 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/66a4c7ba-1a05-47ef-81e8-fb4f93ece49c_isr_1124086_39.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/66a4c7ba-1a05-47ef-81e8-fb4f93ece49c_isr_1124086_39.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 20bf155f20e4293805ab3d0fdb263d36 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/d5e7aa8e-ca98-4738-97a8-941d4da52a38_isr_1123957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/d5e7aa8e-ca98-4738-97a8-941d4da52a38_isr_1123957_18.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash d2fb7e572b322969ec75a539924d37e9 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/42c5c225-eb89-4cd6-9bbc-f97f8693a56e_isr_1124086_41.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/42c5c225-eb89-4cd6-9bbc-f97f8693a56e_isr_1124086_41.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 895155e120e14949284cc8409736aede parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b7338235-7d56-4d18-a814-5336ba831353_isr_1124086_42.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b7338235-7d56-4d18-a814-5336ba831353_isr_1124086_42.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash f4d627265b87f3baeba3b25a211a511d parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9797d463-038d-4168-bc3b-603cecaba8db_isr_1123957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9797d463-038d-4168-bc3b-603cecaba8db_isr_1123957_37.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 63001321a3d37b87adcb85ed4e6f2bfc parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/379603cf-b2d5-435b-8d98-447a3525296c_isr_1123957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/379603cf-b2d5-435b-8d98-447a3525296c_isr_1123957_1.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 0b5e40e2dfd3011df9fa197eff3f4f38 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c2c583f8-9a1b-4c56-b37b-dd86e02d117c_isr_1123953_35.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c2c583f8-9a1b-4c56-b37b-dd86e02d117c_isr_1123953_35.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 073ad50616a1e643c90960b5730fe99a parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba0a732e-3168-45e7-82fd-336de32b8d03_isr_1123957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba0a732e-3168-45e7-82fd-336de32b8d03_isr_1123957_5.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash d93fa6d9c9f388385631b6c990741e00 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/37ba1208-01cb-429d-986f-1e22c1c0bf73_isr_1124086_46.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/37ba1208-01cb-429d-986f-1e22c1c0bf73_isr_1124086_46.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 5bdb1e4ce7a2c646f6c9bd8c9869c8fa parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628089a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/84d6257d-2500-426a-bfc5-ccb8c356511a_isr_1123953_41.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/84d6257d-2500-426a-bfc5-ccb8c356511a_isr_1123953_41.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 07aa4be9a99a31e1d3f7ae8a236a61c5 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7fc2350c-90a8-4d08-ad56-00e133e59c2c_isr_1123955_57.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7fc2350c-90a8-4d08-ad56-00e133e59c2c_isr_1123955_57.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 249389e8177138f6d820fcbb98eb6664 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cb2f9438-0615-4dd6-b4f4-8c0449a19db6_isr_1123957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cb2f9438-0615-4dd6-b4f4-8c0449a19db6_isr_1123957_34.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 5e31453e923a44f6d34f753f29aee0ca parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/290aa6c0-3be5-4c45-af83-1af64982d268_isr_1123953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/290aa6c0-3be5-4c45-af83-1af64982d268_isr_1123953_2.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 5e99c6551e33cd07199f51ca23a3a18e parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/3509bdd7-01eb-4220-9244-89d53b386399_isr_1123957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/3509bdd7-01eb-4220-9244-89d53b386399_isr_1123957_56.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash e79180e648d8ab3d61c500c0c99539fc parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2dbd943e-d115-41b3-acff-854908ba1f1d_isr_1123955_46.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2dbd943e-d115-41b3-acff-854908ba1f1d_isr_1123955_46.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash d3d0a9e4895d7938ca550fba376fd052 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/3f397b55-1c40-4420-bafb-921a6a6883bd_isr_1123955_62.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/3f397b55-1c40-4420-bafb-921a6a6883bd_isr_1123955_62.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 31bf25c0f3c153d9abdb4a44db2ec93b parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38c97d87-4e9b-4f73-ad26-5316f7ae9133_isr_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38c97d87-4e9b-4f73-ad26-5316f7ae9133_isr_1123957_59.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash f0b76be64c8e9d40dc135357b3a1734a parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4ea86fab-3869-4eb8-a956-1709bfdccf0f_isr_1123957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4ea86fab-3869-4eb8-a956-1709bfdccf0f_isr_1123957_32.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 8cf6005bcd28aaccbb24c98d61542f97 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9f66d2e6-3653-40d0-bd10-9bd223476640_isr_1123957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9f66d2e6-3653-40d0-bd10-9bd223476640_isr_1123957_36.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 9d94f6bff79f8ae35068353af2839f3d parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2221bd5a-b167-4674-bfe9-c2ee492ec947_isr_1124086_14.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2221bd5a-b167-4674-bfe9-c2ee492ec947_isr_1124086_14.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 83c1348ef45217264d1f3e818f0a105f parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6f5af050-e260-4757-abd8-16a063141a25_isr_1123953_52.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6f5af050-e260-4757-abd8-16a063141a25_isr_1123953_52.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 6f959bbbc1911812bb9587319511fb22 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2c1fdc04-c68d-4e20-b33e-f4c617451bf5_isr_1124086_21.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2c1fdc04-c68d-4e20-b33e-f4c617451bf5_isr_1124086_21.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 64c8237b2c01f7d473480ae6c843cc7d parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ee4874c2-3991-4c9f-bdc4-5c3f6304dfd1_isr_1123953_48.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ee4874c2-3991-4c9f-bdc4-5c3f6304dfd1_isr_1123953_48.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 3a6b5cb60e8e1a995677730c5821d7fb parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/69ed4e83-e716-4031-8b29-41ad78b1bb7f_isr_1123953_45.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/69ed4e83-e716-4031-8b29-41ad78b1bb7f_isr_1123953_45.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 2c5adacf7b4ba5fc29b869d7dbe6bbc6 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/b74f84e3-40c8-4f75-bef8-c78c383d4962_isr_1123955_21.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/b74f84e3-40c8-4f75-bef8-c78c383d4962_isr_1123955_21.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 9aad89865279dc4a0974f1310d383ea3 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/65601d88-6aac-4815-97b2-1d7d9df2447c_isr_1123953_55.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/65601d88-6aac-4815-97b2-1d7d9df2447c_isr_1123953_55.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 9840f1fac0ab8b8ce11529513d2e5cb9 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3d8cd9eb-2773-4d42-945d-c8d4cc2f732b_isr_1124086_1.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3d8cd9eb-2773-4d42-945d-c8d4cc2f732b_isr_1124086_1.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash e762d5e51c1f989cf15607120ffa15ba parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/09a6bfad-6116-424b-a06b-769040c6b2f4_isr_1123957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/09a6bfad-6116-424b-a06b-769040c6b2f4_isr_1123957_14.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 6bfb40e3c6f62eaa511be94657cd24e5 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/10cf395f-3202-4ab2-a5c6-1e63b2be2fc2_isr_1123955_15.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/10cf395f-3202-4ab2-a5c6-1e63b2be2fc2_isr_1123955_15.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash e1929f1223a701cf75c008505e804a7c parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e01b83df-d980-4030-8544-0f0525c89ba2_isr_1123953_50.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e01b83df-d980-4030-8544-0f0525c89ba2_isr_1123953_50.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 34bae94a2ff3c90e40597de7804f58cc parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/6ab8b6e9-fe2d-4d34-82ea-6a3148074ba1_isr_1124086_59.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/6ab8b6e9-fe2d-4d34-82ea-6a3148074ba1_isr_1124086_59.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash a0cd5b28acb3bc012bd735ce1da356f7 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8378df0c-1c62-4601-8e80-86600ea52d35_isr_1123955_8.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8378df0c-1c62-4601-8e80-86600ea52d35_isr_1123955_8.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 0aff4d2e641e081561c04d4020a07f0e parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c31eb41e-8584-44d5-aa40-4f5d5d3a46a4_isr_1123953_16.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c31eb41e-8584-44d5-aa40-4f5d5d3a46a4_isr_1123953_16.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash dba5b71664f35741d4b1e3e998c474c8 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/281d078c-38fc-41f3-8e5e-11d6a8ec75f4_isr_1123957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/281d078c-38fc-41f3-8e5e-11d6a8ec75f4_isr_1123957_17.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 5ea224b5541043477b2d1bd1766669d4 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/72698028-6738-46af-bf17-d119aaf32000_isr_1123957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/72698028-6738-46af-bf17-d119aaf32000_isr_1123957_9.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 500ce755ff6562b1ca7bc0a3c35ae73f parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7f5a96a3-fb5a-4cb5-87a1-672bf01307d2_isr_1123955_24.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7f5a96a3-fb5a-4cb5-87a1-672bf01307d2_isr_1123955_24.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 996859755a71862a8fa763cab54e6fe2 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/31770d31-0203-4741-bef0-11c88dbb1b87_isr_1123957_8.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/31770d31-0203-4741-bef0-11c88dbb1b87_isr_1123957_8.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 4009486226c348233250041233e67a6a parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/d6bf5932-9a26-4471-aad7-9e05c405d9e1_isr_1124086_13.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/d6bf5932-9a26-4471-aad7-9e05c405d9e1_isr_1124086_13.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 75472cd05bfe37b8c427c304380cc84e parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8188f60b-b563-40be-b63a-c17cd06717bf_isr_1123953_20.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8188f60b-b563-40be-b63a-c17cd06717bf_isr_1123953_20.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 636c1a2c0846a955c0353e2c07a0b1a2 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ae7cfd0b-937c-495d-9142-89d6231ba2d3_isr_1123953_15.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ae7cfd0b-937c-495d-9142-89d6231ba2d3_isr_1123953_15.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash fe85fe44e1be0a7b4fed8100f0640430 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a0561586-68b0-47be-9c56-2440a2eb27d3_isr_1123957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a0561586-68b0-47be-9c56-2440a2eb27d3_isr_1123957_26.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash bb671a63c678c6d89cfff2da9a48f890 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/caf7ccbf-adac-4817-8d30-92147d85cd0b_isr_1124086_29.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/caf7ccbf-adac-4817-8d30-92147d85cd0b_isr_1124086_29.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 6c7385854a5cb719a9e5acdf61012fae parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6b7bfe1f-1378-44aa-a936-ddf18acb888e_isr_1123955_58.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6b7bfe1f-1378-44aa-a936-ddf18acb888e_isr_1123955_58.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 2b6d7befc694e845efba79bbb4b880a7 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/544400de-f09b-4d11-9577-50de62023560_isr_1123953_38.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/544400de-f09b-4d11-9577-50de62023560_isr_1123953_38.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 1ddadf38f495373fbfdec25b3631961e parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2f567505-4b0c-49a7-a327-45355f0140bb_isr_1123955_7.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2f567505-4b0c-49a7-a327-45355f0140bb_isr_1123955_7.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 331ff92742e2f2a46f93b5a1681255e8 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d5fce20c-ca80-468e-a149-60e026aea15c_isr_1123953_10.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d5fce20c-ca80-468e-a149-60e026aea15c_isr_1123953_10.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 324278c33a695fc2dc0fe5b70748915c parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/680f9c63-1265-46b3-8ef4-fc7265724060_isr_1124086_52.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/680f9c63-1265-46b3-8ef4-fc7265724060_isr_1124086_52.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 793834a4097c7d935a532b36a94866d7 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628980e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/0f547878-8926-418f-a03b-629e5f0eb998_isr_1124086_11.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/0f547878-8926-418f-a03b-629e5f0eb998_isr_1124086_11.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash f9aa0385f609f3e200ac3e0ca26affc3 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/883d04e5-f562-46ec-b6bd-976bc99c169c_isr_1124086_37.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/883d04e5-f562-46ec-b6bd-976bc99c169c_isr_1124086_37.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash ecbb8f9a8edc10635f038ff312ce2524 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/97983f87-8eb6-46a2-8cd8-6cd56d84b0f3_isr_1123957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/97983f87-8eb6-46a2-8cd8-6cd56d84b0f3_isr_1123957_35.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 98349ee0bab947d98bf4d2ef7a42b082 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cb7f842-19bd-45dd-80eb-4058cc3fef1d_isr_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cb7f842-19bd-45dd-80eb-4058cc3fef1d_isr_1123955_26.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 488d5d074aa3212f7264659a07457be6 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/009e2acb-0e9d-4010-a696-07f57f8d8dad_isr_1124086_55.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/009e2acb-0e9d-4010-a696-07f57f8d8dad_isr_1124086_55.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash f0aef39a94160888b5e32c2bc1ec539d parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9228ccce-a968-4936-933e-a3720b1e6c5c_isr_1124086_26.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9228ccce-a968-4936-933e-a3720b1e6c5c_isr_1124086_26.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 327dca35edffb1ca41b32a4b2956f395 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628993a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/de6a8285-abf7-4204-a3a6-70d81caf8de2_isr_1123953_26.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/de6a8285-abf7-4204-a3a6-70d81caf8de2_isr_1123953_26.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash d914292e8c580de897872ada0e6db786 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628996c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c5cf59e9-8de7-416c-9b72-cf707c4d22ab_isr_1124086_25.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c5cf59e9-8de7-416c-9b72-cf707c4d22ab_isr_1124086_25.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 7328a30eb273ec07eeb23a5137c43366 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628999e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/3a815154-b921-4035-8193-2d3db4614c8e_isr_1123953_54.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/3a815154-b921-4035-8193-2d3db4614c8e_isr_1123953_54.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 8bdd0168fab50fa9930d3ed77fa9186d parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/14d8dbd9-2b70-436a-a66e-3efced8e02ef_isr_1124086_40.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/14d8dbd9-2b70-436a-a66e-3efced8e02ef_isr_1124086_40.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash c1dc251550528fed8648c334c5721fe3 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/704f4306-e8f6-464f-a99a-c4698fa513ba_isr_1124086_56.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/704f4306-e8f6-464f-a99a-c4698fa513ba_isr_1124086_56.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash a8204efca3be194a66feaa72de90102f parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d8054932-cfb3-479d-a53d-f0704c97c135_isr_1123955_35.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d8054932-cfb3-479d-a53d-f0704c97c135_isr_1123955_35.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash e004e1c162064534001858811bff451d parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a43f5656-7d28-41b2-9005-a9665a4d48b2_isr_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a43f5656-7d28-41b2-9005-a9665a4d48b2_isr_1123957_44.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 2ca50f9fc868030cc04e234f1ffef2f2 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a680af47-7bd4-4377-a9f8-a41f0eb4628b_isr_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a680af47-7bd4-4377-a9f8-a41f0eb4628b_isr_1123953_43.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash a77f7c3c8a4246cc53d2a558cc93abfc parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/890e3c71-0cf9-40c8-afd1-b4ea2a827820_isr_1124086_4.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/890e3c71-0cf9-40c8-afd1-b4ea2a827820_isr_1124086_4.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 667b9ac4f7c94a2bb63eb25644610761 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a4fa284c-c76b-4cfa-a7c0-d50482dc5e8a_isr_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a4fa284c-c76b-4cfa-a7c0-d50482dc5e8a_isr_1123953_23.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 39526a58fa5d3bb35835efb980954141 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f8cbbe59-acaf-4daa-9cbd-945740b790f7_isr_1123955_16.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f8cbbe59-acaf-4daa-9cbd-945740b790f7_isr_1123955_16.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash d51d2460cef4f4e22136ea2751009414 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/e7b8f00b-5daf-4a19-8a89-ea9dcd20a589_isr_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/e7b8f00b-5daf-4a19-8a89-ea9dcd20a589_isr_1123955_4.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash b43c566c4406a23733d07ddf19c829e5 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/387492be-4140-472f-b816-18befdfc650f_isr_1123955_29.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/387492be-4140-472f-b816-18befdfc650f_isr_1123955_29.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 54009002c2e58b1e47a23e1bc67ae82e parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/14070f88-bc91-412d-bfa2-e63b93c3b52b_isr_1123955_1.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/14070f88-bc91-412d-bfa2-e63b93c3b52b_isr_1123955_1.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 9410a0ae5976706fec047e5a2e065f11 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/96b7233c-b0b0-49a1-9261-c4af0d23916c_isr_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/96b7233c-b0b0-49a1-9261-c4af0d23916c_isr_1123953_18.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash ace85c62c89b413c8cb82ee9270149dc parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/cf850369-99f4-4783-995e-b9dda508ba53_isr_1123953_25.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/cf850369-99f4-4783-995e-b9dda508ba53_isr_1123953_25.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash ead77ab8788370c5017b25fcc0457d15 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad18e30f-9692-4290-a19e-9163f802e1f4_isr_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad18e30f-9692-4290-a19e-9163f802e1f4_isr_1123955_56.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 150438d03ff908d6d50007937fd0c94c parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/941afe33-bd9d-4d07-951c-686983ef1646_isr_1123955_30.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/941afe33-bd9d-4d07-951c-686983ef1646_isr_1123955_30.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash dfafdcc26534e34bc30b4e4f4d71e67f parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d1432bda-d40f-452c-9d2f-27968e7cbd12_isr_1123953_14.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d1432bda-d40f-452c-9d2f-27968e7cbd12_isr_1123953_14.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 7813a0500ea1916350f66408edf12608 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f70ae8c4-240c-46c1-9cc9-3e7baab33064_isr_1123953_11.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f70ae8c4-240c-46c1-9cc9-3e7baab33064_isr_1123953_11.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 1e8508e4a661a6fed0331a60304f096b parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/121b615d-324f-42f2-947a-692a2e88571c_isr_1124086_33.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/121b615d-324f-42f2-947a-692a2e88571c_isr_1124086_33.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 667f42530426091c098d21709eaef3ce parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8b410eaf-2103-4a4c-a5bd-05b2d0b00432_isr_1123953_32.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8b410eaf-2103-4a4c-a5bd-05b2d0b00432_isr_1123953_32.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 7bce1bbf7b684948c9b20a7fedf983fb parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/e3b2bbfd-43cc-4075-a6fb-3ea1928172d5_isr_1124086_60.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/e3b2bbfd-43cc-4075-a6fb-3ea1928172d5_isr_1124086_60.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 75202b5e6be856b162823242515b7918 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b57ced11-4505-45c7-8e45-62cec4afa864_isr_1123957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b57ced11-4505-45c7-8e45-62cec4afa864_isr_1123957_12.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 1d94896f515a1d3c9b6fd188260302be parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/58272c75-337d-4eb7-8a6c-5ac8149f8918_isr_1123957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/58272c75-337d-4eb7-8a6c-5ac8149f8918_isr_1123957_55.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash f0b446a2362f3e88371d507c0fc141a7 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/bdc10251-bc2b-4aa2-b3cf-9612fb8ba9a8_isr_1123957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/bdc10251-bc2b-4aa2-b3cf-9612fb8ba9a8_isr_1123957_11.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash eee1ec5622fc264d05e5ccaa50ad370d parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef7d7d38-885e-45d6-98df-2d10fc25c800_isr_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef7d7d38-885e-45d6-98df-2d10fc25c800_isr_1123955_33.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 9b8cd8ef30d3904b4709cddbf0cfd63a parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/af715220-2a51-4fb0-92f7-818e6764c157_isr_1123957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/af715220-2a51-4fb0-92f7-818e6764c157_isr_1123957_49.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 9b4959b2fa84cdc803f631e15dd2efe4 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/53af09b6-5795-4f5c-b5a6-3fca45dc3c10_isr_1123955_38.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/53af09b6-5795-4f5c-b5a6-3fca45dc3c10_isr_1123955_38.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 3c6a6793656dadf698d5299d256c4230 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9661cd3d-ed1e-4399-9806-ddbaca944a18_isr_1123957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9661cd3d-ed1e-4399-9806-ddbaca944a18_isr_1123957_2.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 4cd2ab78c2d679fbd758fb659b0758ff parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/88d9ec60-5358-44db-a4ae-42842adef80e_isr_1124086_2.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/88d9ec60-5358-44db-a4ae-42842adef80e_isr_1124086_2.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 0391e74a134eb15faebbb4cb33fa311c parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5335296d-dc14-4e37-878a-c851cccfd53e_isr_1123957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5335296d-dc14-4e37-878a-c851cccfd53e_isr_1123957_46.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 813d414cb9e6d1ec34afc11dcff6cb01 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/1f544a5e-e822-4091-8e27-d3d476afda8f_isr_1124086_3.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/1f544a5e-e822-4091-8e27-d3d476afda8f_isr_1124086_3.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 2e38b223df25f3fb771ba65705d659cc parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/1732f656-392d-4aa5-a73d-f96a002c51f1_isr_1123955_44.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/1732f656-392d-4aa5-a73d-f96a002c51f1_isr_1123955_44.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 767ad2b70e8f33964cc4fb32f5ea1425 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d0f186d6-3e76-4135-aef4-436e56be5d51_isr_1123955_49.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d0f186d6-3e76-4135-aef4-436e56be5d51_isr_1123955_49.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash b48630f31d664bade3cb1e754f815113 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/53cc511b-8f28-424d-a856-6cbb8b012cbe_isr_1123953_12.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/53cc511b-8f28-424d-a856-6cbb8b012cbe_isr_1123953_12.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash f8424ee4c05bf2ac4c40e3ae8ca3154a parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627284a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5a4f021c-1bca-4014-a083-8ea09cf1de8a_isr_1123955_50.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5a4f021c-1bca-4014-a083-8ea09cf1de8a_isr_1123955_50.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 36e1385d9f83d3b0c3df9e32297ab088 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627287c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2a7063e7-3b59-4415-8b0b-e3ed5d09d692_isr_1124086_24.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2a7063e7-3b59-4415-8b0b-e3ed5d09d692_isr_1124086_24.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 493440d41b7126338978cab869353fb5 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/03919008-a15d-44d9-a868-896712269880_isr_1123953_49.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/03919008-a15d-44d9-a868-896712269880_isr_1123953_49.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 9bc546f32e87aa0328502fe0d8e4f8db parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a63842bd-048c-489f-869c-897c2a25aa2a_isr_1123957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a63842bd-048c-489f-869c-897c2a25aa2a_isr_1123957_21.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 328bd7d49848b354a06d5e4cdcf06936 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8a892656-263f-4934-b3f3-76ae0600cd86_isr_1123955_51.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8a892656-263f-4934-b3f3-76ae0600cd86_isr_1123955_51.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash b11982a23d81dc8860accc296694ee7c parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5035db55-a074-4f01-a75c-8ad541167f39_isr_1124086_45.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5035db55-a074-4f01-a75c-8ad541167f39_isr_1124086_45.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 1d9a16e788274a719c0baa0869bd5c83 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/c692388d-180d-46a2-8514-4c50f1cb5215_isr_1123955_28.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/c692388d-180d-46a2-8514-4c50f1cb5215_isr_1123955_28.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash d9939964aeb31fd592621fda66d108bb parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/887a47fe-fcef-4446-b34c-58507494a503_isr_1124086_35.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/887a47fe-fcef-4446-b34c-58507494a503_isr_1124086_35.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash b997fbfa73235567479b59344b8c4e5b parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a93e2329-5921-4b79-9256-1797ef369405_isr_1123957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a93e2329-5921-4b79-9256-1797ef369405_isr_1123957_60.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 97e39788863405ff0663ee8efaaabd31 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fab45c31-b313-4dcb-932a-01730d15c0cf_isr_1123953_27.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fab45c31-b313-4dcb-932a-01730d15c0cf_isr_1123953_27.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 80b46a2870b59f790522e881e24fe5b2 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/2c79cc35-7f20-446a-a7a2-fee560b9b411_isr_1123957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/2c79cc35-7f20-446a-a7a2-fee560b9b411_isr_1123957_39.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash ab768fc6e796c8676a55c0fd53f6e55e parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3c3eced2-c40b-465b-aa93-3107ecfe5c8c_isr_1124086_17.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3c3eced2-c40b-465b-aa93-3107ecfe5c8c_isr_1124086_17.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash e4f45a139af7d8b446ae14a8caff9772 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/00768c35-8008-44c2-8e15-5450cee27165_isr_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/00768c35-8008-44c2-8e15-5450cee27165_isr_1123955_31.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 3a5c4459cdb8643afba989922da5f0a6 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/7d269cd7-5313-49f0-bfcc-e5a3dddfffd6_isr_1124086_61.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/7d269cd7-5313-49f0-bfcc-e5a3dddfffd6_isr_1124086_61.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash d789422a1ed5c18f3f2a3396f3158b25 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f87aea17-d25a-4a79-aa8a-2fb04364237c_isr_1124086_22.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f87aea17-d25a-4a79-aa8a-2fb04364237c_isr_1124086_22.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 395d876701c5225dbd56dd4d20cca876 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/59274085-eaa3-42db-9dec-4a8d5f4c1097_isr_1123953_58.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/59274085-eaa3-42db-9dec-4a8d5f4c1097_isr_1123953_58.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash fef9d2c09a12f9b725fad47b1b01331e parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fa66b048-6f2d-4101-b790-67318fa3f740_isr_1123953_17.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fa66b048-6f2d-4101-b790-67318fa3f740_isr_1123953_17.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 0055c57241f860768cd58d8ac138ce3e parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9ffd4de9-36b4-41b0-9e65-ff61a24af9f1_isr_1123957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9ffd4de9-36b4-41b0-9e65-ff61a24af9f1_isr_1123957_30.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 03dc67f7577570bb7360329764331ec1 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e955a409-1e33-4d48-a714-9cc5d6612d20_isr_1123957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e955a409-1e33-4d48-a714-9cc5d6612d20_isr_1123957_57.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 49c79daf0860d8930d6c87e3d1ad3d3d parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d9add1a-0f63-47bb-8642-cfda683b11b4_isr_1123953_39.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d9add1a-0f63-47bb-8642-cfda683b11b4_isr_1123953_39.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash da002e15978fa415591432ae3768f80d parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/961d67d4-255c-438f-b86d-13171a2a2b09_isr_1123957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/961d67d4-255c-438f-b86d-13171a2a2b09_isr_1123957_6.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 3b58bee2d0b409450307113441c6ca72 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/357e4691-9037-4188-b01f-13136d918359_isr_1124086_31.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/357e4691-9037-4188-b01f-13136d918359_isr_1124086_31.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 5ae752aa92a117bc5f845923b027521d parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/280ec7f9-1d83-4498-b281-4f3ee6fe92e0_isr_1123957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/280ec7f9-1d83-4498-b281-4f3ee6fe92e0_isr_1123957_23.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash c6835db9d7fde96d209dfbf60deef06c parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f8e4278e-9ecf-426b-8ab0-1eda9779cc20_isr_1123953_60.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f8e4278e-9ecf-426b-8ab0-1eda9779cc20_isr_1123953_60.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 58529606e9d4f20b7131a577a72a06fb parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0dbf2003-a651-4dc8-8162-fdfd2af5a179_isr_1123953_21.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0dbf2003-a651-4dc8-8162-fdfd2af5a179_isr_1123953_21.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash fc19af0e22105d0e46f60ff1ea0af167 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9bcde1f6-548a-4828-9f33-0a7760fd767c_isr_1123955_61.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9bcde1f6-548a-4828-9f33-0a7760fd767c_isr_1123955_61.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 28ba5087eef6b5b983c8209a6a09a28b parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/12b52545-5c67-4a72-9128-5aaba533f450_isr_1123953_19.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/12b52545-5c67-4a72-9128-5aaba533f450_isr_1123953_19.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash ff4f1b132d6ed860c1441de6b7e75b7f parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0e3f851c-cc9c-4599-8444-3914a17e031b_isr_1123953_59.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0e3f851c-cc9c-4599-8444-3914a17e031b_isr_1123953_59.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash a4a659302b774d4120b08792edc95953 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d0bd9e0-e7d7-4796-bc79-0aae515a8b19_isr_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d0bd9e0-e7d7-4796-bc79-0aae515a8b19_isr_1123953_5.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 7f6b0632ea31581f7c98971490b54fc4 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/36cd1b55-c9aa-4023-9b06-15080453aed1_isr_1123955_3.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/36cd1b55-c9aa-4023-9b06-15080453aed1_isr_1123955_3.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash f2ff700ec131fd9af7eef1d7acb91f61 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d3c2c103-4af3-4d7d-9ebb-12a72d6503dc_isr_1123955_34.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d3c2c103-4af3-4d7d-9ebb-12a72d6503dc_isr_1123955_34.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping 2024-03-07 00:38:32 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 61 more blocks parsl.executors.status_handling INFO: Scaling out by 61 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 00:38:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 00:38:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 00:38:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:33 proc_lsst.multi:170 [INFO] [multi] job_id 16882922 proc_lsst.multi INFO: [multi] job_id 16882922 2024-03-07 00:38:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882922 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882922 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16882922 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 00:38:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 00:38:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:33 proc_lsst.multi:170 [INFO] [multi] job_id 16882923 proc_lsst.multi INFO: [multi] job_id 16882923 2024-03-07 00:38:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882923 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882923 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16882923 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 00:38:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 00:38:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:33 proc_lsst.multi:170 [INFO] [multi] job_id 16882924 proc_lsst.multi INFO: [multi] job_id 16882924 2024-03-07 00:38:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882924 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882924 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16882924 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 00:38:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 00:38:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:34 proc_lsst.multi:170 [INFO] [multi] job_id 16882925 proc_lsst.multi INFO: [multi] job_id 16882925 2024-03-07 00:38:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882925 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882925 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16882925 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 00:38:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:34 proc_lsst.multi:170 [INFO] [multi] job_id 16882926 proc_lsst.multi INFO: [multi] job_id 16882926 2024-03-07 00:38:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882926 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882926 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16882926 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 00:38:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:34 proc_lsst.multi:170 [INFO] [multi] job_id 16882927 proc_lsst.multi INFO: [multi] job_id 16882927 2024-03-07 00:38:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 00:38:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882927 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882927 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16882927 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 00:38:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:34 proc_lsst.multi:170 [INFO] [multi] job_id 16882928 proc_lsst.multi INFO: [multi] job_id 16882928 2024-03-07 00:38:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 00:38:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882928 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882928 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16882928 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 00:38:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 00:38:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:35 proc_lsst.multi:170 [INFO] [multi] job_id 16882929 proc_lsst.multi INFO: [multi] job_id 16882929 2024-03-07 00:38:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 00:38:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882929 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882929 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16882929 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 00:38:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 00:38:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 00:38:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:35 proc_lsst.multi:170 [INFO] [multi] job_id 16882931 proc_lsst.multi INFO: [multi] job_id 16882931 2024-03-07 00:38:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 00:38:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882931 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882931 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16882931 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 00:38:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 00:38:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 00:38:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:35 proc_lsst.multi:170 [INFO] [multi] job_id 16882932 proc_lsst.multi INFO: [multi] job_id 16882932 2024-03-07 00:38:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 00:38:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882932 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882932 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16882932 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 00:38:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 00:38:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 00:38:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:36 proc_lsst.multi:170 [INFO] [multi] job_id 16882933 proc_lsst.multi INFO: [multi] job_id 16882933 2024-03-07 00:38:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 00:38:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882933 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882933 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16882933 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 00:38:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:36 proc_lsst.multi:170 [INFO] [multi] job_id 16882934 proc_lsst.multi INFO: [multi] job_id 16882934 2024-03-07 00:38:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 00:38:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882934 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882934 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16882934 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 00:38:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:36 proc_lsst.multi:170 [INFO] [multi] job_id 16882935 proc_lsst.multi INFO: [multi] job_id 16882935 2024-03-07 00:38:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 00:38:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882935 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882935 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16882935 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 00:38:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:36 proc_lsst.multi:170 [INFO] [multi] job_id 16882936 proc_lsst.multi INFO: [multi] job_id 16882936 2024-03-07 00:38:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 00:38:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882936 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882936 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16882936 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 00:38:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 00:38:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882937 proc_lsst.multi INFO: [multi] job_id 16882937 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882937 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882937 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16882937 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882938 proc_lsst.multi INFO: [multi] job_id 16882938 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882938 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882938 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16882938 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882939 proc_lsst.multi INFO: [multi] job_id 16882939 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882939 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882939 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16882939 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882940 proc_lsst.multi INFO: [multi] job_id 16882940 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882940 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882940 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16882940 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882941 proc_lsst.multi INFO: [multi] job_id 16882941 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882941 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882941 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16882941 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882942 proc_lsst.multi INFO: [multi] job_id 16882942 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882942 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882942 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16882942 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882943 proc_lsst.multi INFO: [multi] job_id 16882943 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882943 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882943 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16882943 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882944 proc_lsst.multi INFO: [multi] job_id 16882944 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882944 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882944 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16882944 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882945 proc_lsst.multi INFO: [multi] job_id 16882945 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882945 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882945 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16882945 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:37 proc_lsst.multi:170 [INFO] [multi] job_id 16882946 proc_lsst.multi INFO: [multi] job_id 16882946 2024-03-07 00:38:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 00:38:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882946 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882946 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16882946 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 00:38:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 00:38:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882947 proc_lsst.multi INFO: [multi] job_id 16882947 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882947 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882947 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16882947 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882948 proc_lsst.multi INFO: [multi] job_id 16882948 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882948 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882948 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16882948 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882949 proc_lsst.multi INFO: [multi] job_id 16882949 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882949 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882949 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16882949 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882950 proc_lsst.multi INFO: [multi] job_id 16882950 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882950 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882950 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16882950 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882951 proc_lsst.multi INFO: [multi] job_id 16882951 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882951 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882951 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16882951 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882952 proc_lsst.multi INFO: [multi] job_id 16882952 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16882952 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16882952 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16882952 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882953 proc_lsst.multi INFO: [multi] job_id 16882953 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16882953 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16882953 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16882953 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882954 proc_lsst.multi INFO: [multi] job_id 16882954 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16882954 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16882954 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16882954 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882955 proc_lsst.multi INFO: [multi] job_id 16882955 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16882955 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16882955 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16882955 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:38 proc_lsst.multi:170 [INFO] [multi] job_id 16882956 proc_lsst.multi INFO: [multi] job_id 16882956 2024-03-07 00:38:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16882956 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16882956 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16882956 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-07 00:38:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882957 proc_lsst.multi INFO: [multi] job_id 16882957 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16882957 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16882957 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16882957 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882958 proc_lsst.multi INFO: [multi] job_id 16882958 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882958 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882958 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16882958 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882959 proc_lsst.multi INFO: [multi] job_id 16882959 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882959 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882959 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16882959 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882960 proc_lsst.multi INFO: [multi] job_id 16882960 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882960 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882960 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16882960 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882961 proc_lsst.multi INFO: [multi] job_id 16882961 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882961 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882961 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16882961 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882962 proc_lsst.multi INFO: [multi] job_id 16882962 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882962 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882962 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16882962 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882963 proc_lsst.multi INFO: [multi] job_id 16882963 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882963 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882963 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16882963 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882964 proc_lsst.multi INFO: [multi] job_id 16882964 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882964 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882964 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16882964 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882965 proc_lsst.multi INFO: [multi] job_id 16882965 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882965 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882965 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16882965 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:39 proc_lsst.multi:170 [INFO] [multi] job_id 16882966 proc_lsst.multi INFO: [multi] job_id 16882966 2024-03-07 00:38:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 00:38:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882966 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882966 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16882966 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-07 00:38:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 00:38:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:40 proc_lsst.multi:170 [INFO] [multi] job_id 16882967 proc_lsst.multi INFO: [multi] job_id 16882967 2024-03-07 00:38:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 00:38:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882967 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882967 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16882967 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-07 00:38:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:40 proc_lsst.multi:170 [INFO] [multi] job_id 16882968 proc_lsst.multi INFO: [multi] job_id 16882968 2024-03-07 00:38:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 00:38:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882968 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882968 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16882968 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-07 00:38:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:40 proc_lsst.multi:170 [INFO] [multi] job_id 16882969 proc_lsst.multi INFO: [multi] job_id 16882969 2024-03-07 00:38:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 00:38:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882969 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882969 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16882969 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-07 00:38:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:40 proc_lsst.multi:170 [INFO] [multi] job_id 16882970 proc_lsst.multi INFO: [multi] job_id 16882970 2024-03-07 00:38:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 00:38:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882970 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882970 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16882970 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-07 00:38:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:40 proc_lsst.multi:170 [INFO] [multi] job_id 16882971 proc_lsst.multi INFO: [multi] job_id 16882971 2024-03-07 00:38:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 00:38:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882971 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882971 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16882971 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-07 00:38:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:40 proc_lsst.multi:170 [INFO] [multi] job_id 16882972 proc_lsst.multi INFO: [multi] job_id 16882972 2024-03-07 00:38:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 00:38:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882972 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882972 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16882972 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-07 00:38:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:40 proc_lsst.multi:170 [INFO] [multi] job_id 16882973 proc_lsst.multi INFO: [multi] job_id 16882973 2024-03-07 00:38:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 00:38:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882973 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882973 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16882973 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-07 00:38:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 00:38:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:41 proc_lsst.multi:170 [INFO] [multi] job_id 16882974 proc_lsst.multi INFO: [multi] job_id 16882974 2024-03-07 00:38:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 00:38:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882974 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882974 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16882974 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-07 00:38:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:41 proc_lsst.multi:170 [INFO] [multi] job_id 16882975 proc_lsst.multi INFO: [multi] job_id 16882975 2024-03-07 00:38:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 00:38:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882975 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882975 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16882975 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-07 00:38:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:41 proc_lsst.multi:170 [INFO] [multi] job_id 16882976 proc_lsst.multi INFO: [multi] job_id 16882976 2024-03-07 00:38:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 00:38:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882976 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882976 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16882976 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-07 00:38:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:41 proc_lsst.multi:170 [INFO] [multi] job_id 16882977 proc_lsst.multi INFO: [multi] job_id 16882977 2024-03-07 00:38:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16882977 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16882977 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16882977 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-07 00:38:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:41 proc_lsst.multi:170 [INFO] [multi] job_id 16882978 proc_lsst.multi INFO: [multi] job_id 16882978 2024-03-07 00:38:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16882978 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16882978 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16882978 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-07 00:38:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:42 proc_lsst.multi:170 [INFO] [multi] job_id 16882979 proc_lsst.multi INFO: [multi] job_id 16882979 2024-03-07 00:38:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16882979 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16882979 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16882979 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-07 00:38:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:42 proc_lsst.multi:170 [INFO] [multi] job_id 16882980 proc_lsst.multi INFO: [multi] job_id 16882980 2024-03-07 00:38:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16882980 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16882980 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16882980 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-07 00:38:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:42 proc_lsst.multi:170 [INFO] [multi] job_id 16882981 proc_lsst.multi INFO: [multi] job_id 16882981 2024-03-07 00:38:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16882981 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16882981 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16882981 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-07 00:38:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:42 proc_lsst.multi:170 [INFO] [multi] job_id 16882982 proc_lsst.multi INFO: [multi] job_id 16882982 2024-03-07 00:38:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16882982 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16882982 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16882982 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-07 00:38:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 00:38:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 00:38:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 00:38:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54929 --result_port=54998 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 00:38:42 proc_lsst.multi:170 [INFO] [multi] job_id 16882983 proc_lsst.multi INFO: [multi] job_id 16882983 2024-03-07 00:38:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 00:38:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 00:38:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16882983 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16882983 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16882983 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 043156df027a00d1102ad3a2342a4dfa with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 2424cd0973a1a05fd83524ac06b799cc parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3e6d56cf-2b80-42f8-988d-9a9b9f64002c_isr_1124086_16.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3e6d56cf-2b80-42f8-988d-9a9b9f64002c_isr_1124086_16.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11f94c603985f5af09514a298fa85076 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 44338a6df7e2e5c5fe5e16be33a4e281 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/581163fa-c8a1-4f4e-8a79-ee3577a9c577_isr_1124086_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/581163fa-c8a1-4f4e-8a79-ee3577a9c577_isr_1124086_58.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 57 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20dd0c24ed196a390b56894981c1d229 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 53928c8a6a3b0b685590100addc781f3 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e2f52acd-ac81-41df-9fb7-b3394e9da25c_characterizeImage_1123955_43.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e2f52acd-ac81-41df-9fb7-b3394e9da25c_characterizeImage_1123955_43.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/501f8994-e8b0-451d-91e9-8d9771a12f0b_isr_1123955_43.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/501f8994-e8b0-451d-91e9-8d9771a12f0b_isr_1123955_43.stderr 2024-03-07 00:39:02 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:39:02 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:03 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:03 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:03 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:03 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d922f9eeea1423f9e0d3849a798db74 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 665f7d1c23c3e652128c94b6f6984062 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d7cd7a10-8e9d-4e02-9cdf-740b4a4f6c69_characterizeImage_1123953_36.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d7cd7a10-8e9d-4e02-9cdf-740b4a4f6c69_characterizeImage_1123953_36.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8627d4ed-f210-425a-a4cf-bb7bfeb16d2a_isr_1123953_36.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8627d4ed-f210-425a-a4cf-bb7bfeb16d2a_isr_1123953_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:03 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:04 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:04 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:04 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:04 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:05 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:05 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:05 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:05 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:06 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:06 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:06 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:07 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:07 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:07 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:08 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:08 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:08 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:08 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:08 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:09 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:09 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:09 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:10 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:10 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:10 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:10 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:11 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee0ce2ddc95a4054ea5b559024fe83bc with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash b17bad51a64c33f6365c4ab48604e4d5 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2acb59ba-73d1-43d8-a7e6-7fced8d91e62_characterizeImage_1123953_28.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2acb59ba-73d1-43d8-a7e6-7fced8d91e62_characterizeImage_1123953_28.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/add313a8-e85b-46a8-ac3f-cb453368a488_isr_1123953_28.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/add313a8-e85b-46a8-ac3f-cb453368a488_isr_1123953_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:11 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:11 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:11 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:12 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:12 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:12 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:13 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:13 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:14 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:14 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:14 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:15 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:15 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:15 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:15 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:15 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:16 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:16 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:16 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:16 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:17 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d48595587f5ddfd94e276fd2c94e31b1 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 4cec8d2ee48a4b663cd4d3406679572b parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e055fa9-7c2c-4c6b-b10c-993c57867ec2_characterizeImage_1123955_40.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e055fa9-7c2c-4c6b-b10c-993c57867ec2_characterizeImage_1123955_40.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/45925fd8-d0f5-44c9-9d6f-c1038e0d339e_isr_1123955_40.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/45925fd8-d0f5-44c9-9d6f-c1038e0d339e_isr_1123955_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:17 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:17 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96e71c4dffa07de08ffca5b108fc295e with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash f51fa2699233e9b7ddc9ab4fb3ae0a3e parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35da64ce-8027-439f-a5e5-85a89e8db873_characterizeImage_1123955_41.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35da64ce-8027-439f-a5e5-85a89e8db873_characterizeImage_1123955_41.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6db83033-6bb0-416c-8677-93514b0c4568_isr_1123955_41.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6db83033-6bb0-416c-8677-93514b0c4568_isr_1123955_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:17 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:17 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:18 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:18 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:18 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 81 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 81 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aca7e485cfcbb14899b8e1f92bec6f36 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 3aa738d36669fa89a8c401581fa7c180 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c63ce9e0-9311-4751-8553-4c4e79e69062_characterizeImage_1123955_32.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c63ce9e0-9311-4751-8553-4c4e79e69062_characterizeImage_1123955_32.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef3e10c1-687c-41e8-92b3-7cb24360a933_isr_1123955_32.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef3e10c1-687c-41e8-92b3-7cb24360a933_isr_1123955_32.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ca692df6613658ba6032f9bffab43be with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 91e9f79287f99ecde48c83176c358088 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1facdfad-2470-4a61-95da-031c7515fe6b_characterizeImage_1123955_13.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1facdfad-2470-4a61-95da-031c7515fe6b_characterizeImage_1123955_13.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cbf966c-2fea-4bd4-9548-c4ab8a6f0a21_isr_1123955_13.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cbf966c-2fea-4bd4-9548-c4ab8a6f0a21_isr_1123955_13.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd1ec8bbeb2fd14b139a1cf99160ff0 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash a8d42c7cfd32e556456ab2a7c7b879f5 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5fd72b72-5e17-4a7f-a286-00a19ef9328c_characterizeImage_1123955_9.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5fd72b72-5e17-4a7f-a286-00a19ef9328c_characterizeImage_1123955_9.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f25c9a1d-d09f-4d55-925e-a07738c6cdcf_isr_1123955_9.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f25c9a1d-d09f-4d55-925e-a07738c6cdcf_isr_1123955_9.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a48c636e87828e7dbea4b9728e1612a with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash f991d54fa6ac8ba346cb44d1b4b16758 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/084f2898-9d24-4e5a-a5c8-c8a3c4e0b948_characterizeImage_1123955_11.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/084f2898-9d24-4e5a-a5c8-c8a3c4e0b948_characterizeImage_1123955_11.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/70e27f0d-9e33-4f87-bdaf-eff45e212c5a_isr_1123955_11.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/70e27f0d-9e33-4f87-bdaf-eff45e212c5a_isr_1123955_11.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e8e1294ee3dabe05ca41e7163541e72 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash f5202936de168c8617faffe64655fa2c parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7501502e-23c4-4452-9bcf-c3e5ecb3ed63_characterizeImage_1123953_42.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7501502e-23c4-4452-9bcf-c3e5ecb3ed63_characterizeImage_1123953_42.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea4f46f1-85a5-4476-984d-3d4e76eb4cc1_isr_1123953_42.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea4f46f1-85a5-4476-984d-3d4e76eb4cc1_isr_1123953_42.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d380bd4c596f0d5bf18477e2f3f1ae1 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash ccac0af4999148de8121aeccf4aafa16 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/13115233-2acc-4c86-85fe-15976760f33e_characterizeImage_1123955_5.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/13115233-2acc-4c86-85fe-15976760f33e_characterizeImage_1123955_5.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/545b81a6-dbc0-4319-bb73-e7c542fd87de_isr_1123955_5.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/545b81a6-dbc0-4319-bb73-e7c542fd87de_isr_1123955_5.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11f127693f59f5f840fdebdb3fc0d7cb with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 718fb1e11434d7685895ba549470d5d6 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/9c5d064b-9329-4287-ad73-a2dfca89743d_characterizeImage_1123953_53.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/9c5d064b-9329-4287-ad73-a2dfca89743d_characterizeImage_1123953_53.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/2464b019-443b-40a7-b470-f42bd9e4eb8a_isr_1123953_53.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/2464b019-443b-40a7-b470-f42bd9e4eb8a_isr_1123953_53.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b67b550fc0bc20c717a5bbfcf99ec7c with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 8a6906477b5339ad7873a35dd6fc0ded parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0bbfb682-0e2d-4951-b8a6-b896fad31e6e_characterizeImage_1123957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0bbfb682-0e2d-4951-b8a6-b896fad31e6e_characterizeImage_1123957_54.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e568a80e-8c97-4532-89b6-9ee520a406ef_isr_1123957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e568a80e-8c97-4532-89b6-9ee520a406ef_isr_1123957_54.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2be62b5ea9a5939135fac3b8b758e71a with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash c7bae79fa12181cd0eb69565daf465d6 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/25311660-4478-4873-b34d-facacf7b54c3_characterizeImage_1123957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/25311660-4478-4873-b34d-facacf7b54c3_characterizeImage_1123957_29.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/227c0efa-509a-42ad-baad-60f92f25cadc_isr_1123957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/227c0efa-509a-42ad-baad-60f92f25cadc_isr_1123957_29.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb85f4462636768740cb6a883584dd2 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 07b690fed1429b41a246d116eddbd89e parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c19ff0ec-81ab-4653-a8cc-365952216c1c_characterizeImage_1123953_3.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c19ff0ec-81ab-4653-a8cc-365952216c1c_characterizeImage_1123953_3.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/97ab3281-73ab-41c8-b11a-d92d62c9a1bb_isr_1123953_3.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/97ab3281-73ab-41c8-b11a-d92d62c9a1bb_isr_1123953_3.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab35e444ff58c1e029a9a9f557b13141 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 50f14de27581409ac7c61f02962b9d8a parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a26e811b-2fa5-4dbf-b2bb-fe59c2f5e656_characterizeImage_1123953_40.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a26e811b-2fa5-4dbf-b2bb-fe59c2f5e656_characterizeImage_1123953_40.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/509fcdcf-5b31-4f5c-b1b5-54ee779715b0_isr_1123953_40.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/509fcdcf-5b31-4f5c-b1b5-54ee779715b0_isr_1123953_40.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24a7edf66c3f8c3e06d64cf8017c21f6 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 22df70d5f0dfadde174606e652bcd90d parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63906ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/34831632-7e23-4379-8b6b-d3fb1a569511_characterizeImage_1123955_53.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/34831632-7e23-4379-8b6b-d3fb1a569511_characterizeImage_1123955_53.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/885dae52-e8e6-4047-9eee-82b355f71dc7_isr_1123955_53.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/885dae52-e8e6-4047-9eee-82b355f71dc7_isr_1123955_53.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d19d2bde2222e71dc6f7fcda17eb2e with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 514cc06830849b05bc3ad2c9a3f4fc1f parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3798d24b-1094-4cdf-8eb2-52d8c67e36cc_characterizeImage_1123955_10.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3798d24b-1094-4cdf-8eb2-52d8c67e36cc_characterizeImage_1123955_10.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7356a218-b4e5-40f0-8332-0688c6f1608c_isr_1123955_10.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7356a218-b4e5-40f0-8332-0688c6f1608c_isr_1123955_10.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5825613a12f6e91ea9d85ee88114e344 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash fd5e60ca0499ebf65b2043b1bb9c1455 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9a293480-7701-4dd5-bd8b-79026dc2d983_isr_1124086_19.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9a293480-7701-4dd5-bd8b-79026dc2d983_isr_1124086_19.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f37c12fa5b064913c5989bab33becc20 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash b86abb6c62c2d790e33f29b8aa34b757 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c3c1caf4-ce99-4639-8ca2-888c1baafb28_characterizeImage_1123953_8.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c3c1caf4-ce99-4639-8ca2-888c1baafb28_characterizeImage_1123953_8.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/bb83d0d6-8e21-4663-8f0e-d336b97c5d32_isr_1123953_8.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/bb83d0d6-8e21-4663-8f0e-d336b97c5d32_isr_1123953_8.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93482d01f8607980fde192fefc201933 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 6cc62610edffd82c53c3d92f459aa71c parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ca3ee79e-e3ff-400c-9be4-3090244d14f1_characterizeImage_1123955_59.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ca3ee79e-e3ff-400c-9be4-3090244d14f1_characterizeImage_1123955_59.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f0d9df33-2f3e-46b0-967d-d054045d72b9_isr_1123955_59.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f0d9df33-2f3e-46b0-967d-d054045d72b9_isr_1123955_59.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62c6f3bba935aab64f27166365d25333 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 4087e0d31e0768cf60a9c40165f52e84 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/415290e1-e167-4e15-99ae-7729d18e45cd_characterizeImage_1123953_13.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/415290e1-e167-4e15-99ae-7729d18e45cd_characterizeImage_1123953_13.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ad05ab9f-ad6c-493a-9145-5c4c54a3a65a_isr_1123953_13.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ad05ab9f-ad6c-493a-9145-5c4c54a3a65a_isr_1123953_13.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a492538a905c2df27ae4202c0efe53 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash d2e636690f5093be301a212179109e78 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63906d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe0cfc1-802d-4465-a85f-6a51b887c6e2_characterizeImage_1123957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe0cfc1-802d-4465-a85f-6a51b887c6e2_characterizeImage_1123957_53.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b224c961-6136-4a8b-ba17-4b70bbd03b9d_isr_1123957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b224c961-6136-4a8b-ba17-4b70bbd03b9d_isr_1123957_53.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd2d4db0e9d8bd841d5776a109eb4adb with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash b7b344dff6bdb8ddb854c8312c5e7d46 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/6a997b25-bd59-4c3f-a27c-94cad633d996_characterizeImage_1123955_37.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/6a997b25-bd59-4c3f-a27c-94cad633d996_characterizeImage_1123955_37.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad6ec155-d552-4999-8fdd-55922c978386_isr_1123955_37.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad6ec155-d552-4999-8fdd-55922c978386_isr_1123955_37.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9102fc7bcedce4393f646f790be3cb75 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 30ac0dc9cef3a5944923cc16a1da9084 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8f113fa1-91b5-4ae0-8477-0acdbe840a32_characterizeImage_1123953_57.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8f113fa1-91b5-4ae0-8477-0acdbe840a32_characterizeImage_1123953_57.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a8e6cede-ebfe-4cb6-8c4a-86b7ec508a91_isr_1123953_57.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a8e6cede-ebfe-4cb6-8c4a-86b7ec508a91_isr_1123953_57.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1adbdfe46a251855d16183d7fbac1716 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash ca5e1a1ee94b8e173b279fbcd3986adf parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2fedd920-f4cc-45e1-9e1f-e5ddcef86912_isr_1124086_51.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2fedd920-f4cc-45e1-9e1f-e5ddcef86912_isr_1124086_51.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fe5f5302f5a9cc32889cb934a5c6b60 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash af8f3b018eb2730835430d25b15b9630 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5a59b231-f9d6-4ccc-b5ee-675f9a44ab82_isr_1124086_8.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5a59b231-f9d6-4ccc-b5ee-675f9a44ab82_isr_1124086_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 81 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95fdd8dd4e0f2502c9fd3f530f3e77c0 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 2e17cc24ee39e5b65e2951f1cd5142dd parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2f6bded2-aa6c-4fab-96ff-78c0be943b47_isr_1124086_62.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2f6bded2-aa6c-4fab-96ff-78c0be943b47_isr_1124086_62.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fff401ab8e83dfd4c7b54d69f01343f5 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 66a5c7d65ac0913a274af22ce100e4c6 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/afe1ab77-7f0d-4019-b96c-6add22d747d8_isr_1124086_44.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/afe1ab77-7f0d-4019-b96c-6add22d747d8_isr_1124086_44.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6200c58dd5525978a5028e2588273cd with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 804ae82d5ae294d546a262a0813a41fa parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bbd531ba-d412-49c3-a9f3-27b34e690a79_isr_1124086_47.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bbd531ba-d412-49c3-a9f3-27b34e690a79_isr_1124086_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 81 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e0c4e2c1e3a73c2025e9cc196c0fa88 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 3474378a0deb6d23ff4a7e1a049e3e1e parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c2e8c105-d4f2-45f5-bddb-36e6b3923790_characterizeImage_1123955_27.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c2e8c105-d4f2-45f5-bddb-36e6b3923790_characterizeImage_1123955_27.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/fac09580-f172-4d94-af8d-bfab634c3611_isr_1123955_27.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/fac09580-f172-4d94-af8d-bfab634c3611_isr_1123955_27.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1e6846f056e92e1fe1b9c8934ba0c90 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 440e0b0fbd9f1bb20ad74b53942c82b4 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64fe7f59-3d65-441a-b012-9835b2961eb3_characterizeImage_1123955_60.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64fe7f59-3d65-441a-b012-9835b2961eb3_characterizeImage_1123955_60.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d2d98d39-5c7d-4833-889c-86b94daa291c_isr_1123955_60.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d2d98d39-5c7d-4833-889c-86b94daa291c_isr_1123955_60.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b285857071561c66acad8b169094359a with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 65f59b805aae2342559d4fcd5e2e07da parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a6c9c2ae-bfce-40c0-9793-fa47a388637d_characterizeImage_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a6c9c2ae-bfce-40c0-9793-fa47a388637d_characterizeImage_1123957_31.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f12cc5f9-291a-40cc-8259-dee9b4ca1c6d_isr_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f12cc5f9-291a-40cc-8259-dee9b4ca1c6d_isr_1123957_31.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb3d86e5068b654a1110388db8584825 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 5e62f4e94db37e716d090744a577c3bb parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb2b916a-50ca-42ed-be8a-8b77d0a36650_characterizeImage_1123953_46.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb2b916a-50ca-42ed-be8a-8b77d0a36650_characterizeImage_1123953_46.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/5d92111a-3e46-441c-84f0-83fc07dbda13_isr_1123953_46.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/5d92111a-3e46-441c-84f0-83fc07dbda13_isr_1123953_46.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbd5be11e100730fa8515f263cf7808c with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 352eb8e94612665e98f6a5cec527867f parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3f2c6947-429c-4c91-9eef-d44d2d69c40b_characterizeImage_1123953_6.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3f2c6947-429c-4c91-9eef-d44d2d69c40b_characterizeImage_1123953_6.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/55adb9dd-15d6-4836-a085-350ec9cbf937_isr_1123953_6.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/55adb9dd-15d6-4836-a085-350ec9cbf937_isr_1123953_6.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06324bf5c4cf133c0484cd4d79e19bba with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash b44e6c05bc55bd8484d2be3ec1d83ba6 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c9fb6a76-62b4-4092-9f46-ce288b155512_characterizeImage_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c9fb6a76-62b4-4092-9f46-ce288b155512_characterizeImage_1123953_61.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0a194842-94c5-4b60-a67d-90d15c4abff2_isr_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0a194842-94c5-4b60-a67d-90d15c4abff2_isr_1123953_61.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53b9c9b0ab6b6c4de378be6cc0163d25 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 3afbb0e63e8066cd1e9037bdfbf3b59e parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/24ef8443-c405-49ee-8e6a-721d05fbeaaf_characterizeImage_1123953_33.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/24ef8443-c405-49ee-8e6a-721d05fbeaaf_characterizeImage_1123953_33.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a87a2d74-4e2e-41ff-8b23-dae7a6393b11_isr_1123953_33.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a87a2d74-4e2e-41ff-8b23-dae7a6393b11_isr_1123953_33.stderr 2024-03-07 00:39:33 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:39:33 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:34 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:34 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:34 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:34 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:34 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63829d248e85ddd62157f31483c53d42 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 3ba6c4b4aeab6b804751c7961a157140 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:35 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c9efe260-ff2f-435e-9835-bd69c572aec3_characterizeImage_1123957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c9efe260-ff2f-435e-9835-bd69c572aec3_characterizeImage_1123957_20.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6b00b6e9-4f9d-41fa-a50a-d0907247f3ac_isr_1123957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6b00b6e9-4f9d-41fa-a50a-d0907247f3ac_isr_1123957_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:35 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d3a6768af597244ecc9c8a519c9fa2 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash da4ced1143be1785788d565253a703af parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:35 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/dfeee673-d793-4dc8-8e83-7dbe46431492_isr_1124086_28.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/dfeee673-d793-4dc8-8e83-7dbe46431492_isr_1124086_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:35 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:35 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:35 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:36 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:36 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:36 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8da1d0c85b3c02f88204d556ab4feddf with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash de398aab062d48318bf974200a455e07 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6e0527d5-63bc-4083-aa6e-15e2e1dac337_characterizeImage_1123957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6e0527d5-63bc-4083-aa6e-15e2e1dac337_characterizeImage_1123957_51.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba6938ef-895a-44c9-a071-3512e5872cec_isr_1123957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba6938ef-895a-44c9-a071-3512e5872cec_isr_1123957_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:37 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9ae8d8055e63acafc60a2e6476a3904 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 57204c40005aedc45d3730f350fd0a55 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c363dfa6-e6d6-48b1-b013-69191048b580_isr_1124086_5.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c363dfa6-e6d6-48b1-b013-69191048b580_isr_1124086_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:37 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91cd330df7866fda6852a93e6359ae86 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 26490521d4c66baa743e9d5109da36c6 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90cc17ed-6e38-4a11-b10c-2a20b4a8c371_characterizeImage_1123957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90cc17ed-6e38-4a11-b10c-2a20b4a8c371_characterizeImage_1123957_42.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b8d50c64-a75a-4f4f-8f13-5182ac3538ac_isr_1123957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b8d50c64-a75a-4f4f-8f13-5182ac3538ac_isr_1123957_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:37 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab7e411644fcfc9f18b101946bd2794b with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash ab8e68b25cac6435efb0da8a1f3c16d1 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/340106fb-a8cd-403c-a58e-bde03cb5fef6_characterizeImage_1123957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/340106fb-a8cd-403c-a58e-bde03cb5fef6_characterizeImage_1123957_43.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f97e29f6-b586-4f10-841f-50f0ffe77a94_isr_1123957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f97e29f6-b586-4f10-841f-50f0ffe77a94_isr_1123957_43.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99895d7d6d49eb894c0b48f5b18a1978 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 8cb322a28126ff984bce96bfb1d948df parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/af0e8036-1469-4ae5-be31-216897612814_characterizeImage_1123955_14.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/af0e8036-1469-4ae5-be31-216897612814_characterizeImage_1123955_14.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7cf81efa-6133-4f52-9194-ff317b7390ca_isr_1123955_14.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7cf81efa-6133-4f52-9194-ff317b7390ca_isr_1123955_14.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d92cd47e512086bb21bb7f8637abd24 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 3817f26de0617b8135932257eadf4654 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/07a21969-25c7-4d1e-9853-b3a9f79bf309_characterizeImage_1123957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/07a21969-25c7-4d1e-9853-b3a9f79bf309_characterizeImage_1123957_62.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/fb2d698b-2d10-42b5-9af7-e75a5b5c558a_isr_1123957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/fb2d698b-2d10-42b5-9af7-e75a5b5c558a_isr_1123957_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:38 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29175837f0dc74f3589078eed6212a75 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 5d93694a3014b23863064ab4d6357060 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e0a514f6-9ba7-43a5-b958-134c15960edd_characterizeImage_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e0a514f6-9ba7-43a5-b958-134c15960edd_characterizeImage_1123955_42.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/63bcc597-5b01-4854-b732-20264eb940e8_isr_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/63bcc597-5b01-4854-b732-20264eb940e8_isr_1123955_42.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a845e633d894c9ed79d28e78b51e42ae with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 977948b6d97373167b64a58a7610d16c parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/90ca82b6-d60d-480e-befe-976c81cde2a7_isr_1124086_30.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/90ca82b6-d60d-480e-befe-976c81cde2a7_isr_1124086_30.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbf8ff1b53cf76006fb87827b062d5f4 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash a1b0f9a2d8c46eb08134af70849f44b1 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/24d59b2b-68d0-4d08-b3d5-0f5e6a771b3c_isr_1124086_10.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/24d59b2b-68d0-4d08-b3d5-0f5e6a771b3c_isr_1124086_10.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3531d8c7125cbc747ac60700b7cd08b5 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash f9096c5fb8acc4699c470f9e8ef75b67 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2f8812e2-8596-4acf-b252-a8db3480e925_characterizeImage_1123955_12.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2f8812e2-8596-4acf-b252-a8db3480e925_characterizeImage_1123955_12.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a6ff9c76-0c4f-4d2f-aad8-4b7b60d101ab_isr_1123955_12.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a6ff9c76-0c4f-4d2f-aad8-4b7b60d101ab_isr_1123955_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:38 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:38 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:39 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e901e508de63e380becd306ccd9482b3 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 2068c067b9ee7d437379b30f03dff674 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639acd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/4c5c1725-aba8-4257-b381-91c77aaa542d_characterizeImage_1123955_36.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/4c5c1725-aba8-4257-b381-91c77aaa542d_characterizeImage_1123955_36.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8ce1abe8-da2b-4362-87cf-0ae1fb6f8008_isr_1123955_36.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8ce1abe8-da2b-4362-87cf-0ae1fb6f8008_isr_1123955_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:39 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:39 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:40 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d29c057d2d5be8fa49f8b7982905d8b with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash f2d6e8b33b62550cd6ca9feefa25bf06 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/50ac643d-587f-4cdb-b454-d7033ecaa338_isr_1123955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/50ac643d-587f-4cdb-b454-d7033ecaa338_isr_1123955_2.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c26c2991fb8acfaf59966ba7b09716c with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 29da1a067e194106f9ab7ba7fbeeb7ed parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1b1483a1-7118-414e-b47f-c2a187935ea0_characterizeImage_1123953_4.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1b1483a1-7118-414e-b47f-c2a187935ea0_characterizeImage_1123953_4.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/73adf95c-d633-4d7a-aa3d-bb56a851cf26_isr_1123953_4.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/73adf95c-d633-4d7a-aa3d-bb56a851cf26_isr_1123953_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:40 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35b2a987fe1fd25c78adfd200dc876ce with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash c95dc80d011ad5fa5b302e04c0483d46 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639628e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/87bc1aa0-e419-43af-9f3d-3cc732ee252b_characterizeImage_1123957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/87bc1aa0-e419-43af-9f3d-3cc732ee252b_characterizeImage_1123957_33.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6f48242d-33b2-4cd9-9263-b21e821aab3e_isr_1123957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/6f48242d-33b2-4cd9-9263-b21e821aab3e_isr_1123957_33.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8595f1867b9a9a9d767d73184e84b6b with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash bb35ffbcd89790a85fc7a79a143f5433 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/9082b2c2-4cde-405a-9d2f-8a654059ac08_characterizeImage_1123957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/9082b2c2-4cde-405a-9d2f-8a654059ac08_characterizeImage_1123957_15.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/0df06c2a-7a4f-4f31-942e-820971a360e8_isr_1123957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/0df06c2a-7a4f-4f31-942e-820971a360e8_isr_1123957_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:40 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a26a6afac0efa3460e9d43ae6e920d4d with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 6fb53547b2a99fc9dde58c38ff5fd974 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/da997a21-7397-495a-afb0-168b2dbc61e1_characterizeImage_1123957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/da997a21-7397-495a-afb0-168b2dbc61e1_characterizeImage_1123957_41.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1fdf551e-60d9-4576-bcd5-d34fa9363ec7_isr_1123957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1fdf551e-60d9-4576-bcd5-d34fa9363ec7_isr_1123957_41.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f0f2d8a60700913d602e4023876321b with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash cd57927a6d29538e25cd3858b9d67bfe parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d2242eed-bbd2-4536-b223-23126326bb3a_characterizeImage_1123955_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d2242eed-bbd2-4536-b223-23126326bb3a_characterizeImage_1123955_23.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/890cb3a2-eea6-40e9-9c0c-31271d7851c2_isr_1123955_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/890cb3a2-eea6-40e9-9c0c-31271d7851c2_isr_1123955_23.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d47a0c40f2f04bb575a319322bde138 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash ada16985ce8dc4386a7ad5f4f674ba2d parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d05c3d7b-9649-4e94-aae8-bd804484786c_characterizeImage_1123953_24.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d05c3d7b-9649-4e94-aae8-bd804484786c_characterizeImage_1123953_24.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c4bfa857-0b49-4139-a900-c4cb66f6c7ba_isr_1123953_24.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c4bfa857-0b49-4139-a900-c4cb66f6c7ba_isr_1123953_24.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7213e4b9ad5998da4ee9268c0c491f08 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 18862d370194de3a4c0a37ed109c4356 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64e9ceaa-ecb2-4edb-bf41-79615b53d5ad_characterizeImage_1123955_48.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64e9ceaa-ecb2-4edb-bf41-79615b53d5ad_characterizeImage_1123955_48.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/0e13b284-1e2b-4560-8206-279062ea06a1_isr_1123955_48.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/0e13b284-1e2b-4560-8206-279062ea06a1_isr_1123955_48.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e00a20d9373328b6147c34bdc978daa with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 01ab0ff6add0cb29fe2ee623a7c3d94a parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f2c52014-bda5-47e7-a7b7-92b252872e09_characterizeImage_1123955_25.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f2c52014-bda5-47e7-a7b7-92b252872e09_characterizeImage_1123955_25.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/84cb695b-5891-4ce4-ae16-70c9cc511a72_isr_1123955_25.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/84cb695b-5891-4ce4-ae16-70c9cc511a72_isr_1123955_25.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7c38ed3466399ffa158dca4c2127291 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 954d0240d2b2e2c109e45382c3a20fcf parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/34cb97ce-2df8-4ccc-bd3c-a5d467b3d484_characterizeImage_1123957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/34cb97ce-2df8-4ccc-bd3c-a5d467b3d484_characterizeImage_1123957_7.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f2f82aa1-8e5d-431d-94c8-77d22cf49018_isr_1123957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/f2f82aa1-8e5d-431d-94c8-77d22cf49018_isr_1123957_7.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a79105beea3bfdb6175e0303b9ca7564 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash b30c485162c382cf68a4ee2e07242f98 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/78f640b5-9e5b-43c2-8ae5-89b8c71c4155_characterizeImage_1123953_30.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/78f640b5-9e5b-43c2-8ae5-89b8c71c4155_characterizeImage_1123953_30.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f6f66d12-7162-4fb2-b611-49f235ad6b94_isr_1123953_30.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f6f66d12-7162-4fb2-b611-49f235ad6b94_isr_1123953_30.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 117a6bc7ba172173dffe815bcd02a27a with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 4de5fe051e17886edb4167539ec29d15 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639609a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/41acb19b-9da1-4752-8925-0cb41b0574d4_characterizeImage_1123955_22.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/41acb19b-9da1-4752-8925-0cb41b0574d4_characterizeImage_1123955_22.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4159ac2f-f295-4751-bb15-72aaef87373c_isr_1123955_22.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4159ac2f-f295-4751-bb15-72aaef87373c_isr_1123955_22.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac714661946f13bb2ba0b936076888d5 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash c20374f686dff73457fa0bf320e2a6e1 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/02d31865-d1d5-4a90-818a-438ccd63cfef_characterizeImage_1123955_39.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/02d31865-d1d5-4a90-818a-438ccd63cfef_characterizeImage_1123955_39.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d6284207-b9b4-479e-a137-50cf03bc4c5c_isr_1123955_39.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d6284207-b9b4-479e-a137-50cf03bc4c5c_isr_1123955_39.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d816b163ed57e7b03fe4c47c4a74dc59 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 1f7b5a60331c086cf06ceed9750c6504 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/de5dfd98-6ba8-4406-91b8-cc18df0150cb_isr_1124086_9.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/de5dfd98-6ba8-4406-91b8-cc18df0150cb_isr_1124086_9.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8dbf1f666c1e06f764aee903f74ca5c with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 570769d5c76ec82bcac0109a091b40ca parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/758a6ace-ff2d-4ee3-bd29-c07fda6e6aae_isr_1124086_54.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/758a6ace-ff2d-4ee3-bd29-c07fda6e6aae_isr_1124086_54.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 571e4a9c433883ed5d3e37fe02938789 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash c21c21a765de970dcf784e8991372094 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/42f4adb4-20d1-4d75-8c3e-dc0651603f53_characterizeImage_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/42f4adb4-20d1-4d75-8c3e-dc0651603f53_characterizeImage_1123953_31.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e5dab166-5351-4acc-84d3-da68ed102290_isr_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e5dab166-5351-4acc-84d3-da68ed102290_isr_1123953_31.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69c7efa4aba4b47f4617a1cb418e5693 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 3afc849b2c8e09e2de489bd5038b7daa parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/94938c59-63b7-4fee-8b6d-d30902073539_characterizeImage_1123953_37.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/94938c59-63b7-4fee-8b6d-d30902073539_characterizeImage_1123953_37.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/15939763-0fda-4c71-83bf-fc207666c5f6_isr_1123953_37.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/15939763-0fda-4c71-83bf-fc207666c5f6_isr_1123953_37.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67e4cb015024ac28589276f1e12e1327 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash c79745b3527d1ca62b56280c338a837a parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8e3cdf69-c54c-4b4d-b041-6d422dcc2e40_characterizeImage_1123953_9.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8e3cdf69-c54c-4b4d-b041-6d422dcc2e40_characterizeImage_1123953_9.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/4d21185a-ac7e-44b1-a9e8-3c4607ddc406_isr_1123953_9.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/4d21185a-ac7e-44b1-a9e8-3c4607ddc406_isr_1123953_9.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e38624d56e0208f78b5ffceffc2a930 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 0d07ae8851050377f00e2af272bc9c2e parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bf45ff94-954d-4110-9942-efa871d4f08f_characterizeImage_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bf45ff94-954d-4110-9942-efa871d4f08f_characterizeImage_1123957_61.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/918c0595-5c5a-40f1-ace8-3e0ad51201e9_isr_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/918c0595-5c5a-40f1-ace8-3e0ad51201e9_isr_1123957_61.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc137a4648ff23790b22a35bdc6ad07 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 9b1aed6ee546021ff4961dd3ea47740f parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f505d930-556f-4df1-a196-1398c820b952_characterizeImage_1123957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f505d930-556f-4df1-a196-1398c820b952_characterizeImage_1123957_16.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/28a9ad41-2518-4a2c-b2a7-2e498c16d883_isr_1123957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/28a9ad41-2518-4a2c-b2a7-2e498c16d883_isr_1123957_16.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51d8e56b1c05a061c3ccc219bedb8823 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 49029c67ffec79a18d4f5cd92e2cdcf9 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/cc490b81-e030-4f98-9381-4e564f8449ea_characterizeImage_1123957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/cc490b81-e030-4f98-9381-4e564f8449ea_characterizeImage_1123957_19.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5619e658-549b-4360-89cd-1ed90a2e5623_isr_1123957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5619e658-549b-4360-89cd-1ed90a2e5623_isr_1123957_19.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48ab3e8635073de353fa0c2fc17d41f0 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 47a3981b36b1a24914596af76d99ec39 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ff423ae4-fae2-4940-b577-cf353ea6de18_characterizeImage_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ff423ae4-fae2-4940-b577-cf353ea6de18_characterizeImage_1123955_19.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/af3ba71f-b1ae-451a-aef8-4b9bbfb96880_isr_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/af3ba71f-b1ae-451a-aef8-4b9bbfb96880_isr_1123955_19.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ab6b36c4b8c2dd9371627c433eb27ec with result from task 87 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:40 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash d3aecd33397369753ed733fc71c10ec2 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bd8e13-b190-4f25-9d3b-c6762590da44_characterizeImage_1123955_47.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bd8e13-b190-4f25-9d3b-c6762590da44_characterizeImage_1123955_47.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/722e649f-7d53-4544-add4-d2e37d4ddebb_isr_1123955_47.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/722e649f-7d53-4544-add4-d2e37d4ddebb_isr_1123955_47.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aee641025521512a4761fb055325dd37 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 7b6a8b42a61cc27894e92b39e6ead94f parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/dd22ef8c-9a78-492c-81ae-e4b474a8daef_characterizeImage_1123957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/dd22ef8c-9a78-492c-81ae-e4b474a8daef_characterizeImage_1123957_52.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/12a01a31-a1bc-4cc2-baf8-e9ebc76ca3ac_isr_1123957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/12a01a31-a1bc-4cc2-baf8-e9ebc76ca3ac_isr_1123957_52.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abcddd369c6b5e6da311432153290124 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 9132c49ae6b966707a495e215314c8e0 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe57bfb-3e40-4587-9478-56946e34f661_characterizeImage_1123957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe57bfb-3e40-4587-9478-56946e34f661_characterizeImage_1123957_3.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e6d89b66-6a8b-476c-8264-876f03ec05fb_isr_1123957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e6d89b66-6a8b-476c-8264-876f03ec05fb_isr_1123957_3.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f30c801b33d7828722cd6db9439948e2 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 52e3fd16bd6f4119e0da51d2452d97e2 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2a543a3b-df48-4998-93bb-8a7b2ad48f47_characterizeImage_1123953_44.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2a543a3b-df48-4998-93bb-8a7b2ad48f47_characterizeImage_1123953_44.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d3528af1-da1a-4be0-9b1c-da1be68f8265_isr_1123953_44.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d3528af1-da1a-4be0-9b1c-da1be68f8265_isr_1123953_44.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b76182796ca4c2027c3da41dd4dbd33 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 7fe2f5d70b6fb69f08693249dc78b057 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/89797aa6-4662-4a0a-a0f4-f8758c19fb3c_characterizeImage_1123957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/89797aa6-4662-4a0a-a0f4-f8758c19fb3c_characterizeImage_1123957_45.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/321d21a6-3ed7-46b0-b2c3-e85195bfebc9_isr_1123957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/321d21a6-3ed7-46b0-b2c3-e85195bfebc9_isr_1123957_45.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74807a5a0a02a33c34038b5b387b750c with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash b34a0bb75ddd9d6447b8887a9393d9ba parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f7b0ac41-5758-44bc-8d26-133b6cb3bb28_isr_1124086_57.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f7b0ac41-5758-44bc-8d26-133b6cb3bb28_isr_1124086_57.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d21b770ad5ecff85ff6da081ed397ca0 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 638cfeddb262ffdabf048ece5d9fdfdb parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e4a80988-19dd-44b4-b40b-04ea2abacfe4_characterizeImage_1123953_62.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e4a80988-19dd-44b4-b40b-04ea2abacfe4_characterizeImage_1123953_62.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/999ec973-32f1-4024-a44a-2127c724a03e_isr_1123953_62.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/999ec973-32f1-4024-a44a-2127c724a03e_isr_1123953_62.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aafad119d021ebb7cd30c4aa5c5a273 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 73e70045573d8229e55d595337359440 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8071d203-70e1-4f68-ac0e-6fc117a75477_characterizeImage_1123955_17.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8071d203-70e1-4f68-ac0e-6fc117a75477_characterizeImage_1123955_17.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2b534a1f-3d71-494c-b225-17a8429a9f87_isr_1123955_17.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2b534a1f-3d71-494c-b225-17a8429a9f87_isr_1123955_17.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fbd2d27ce811221fdbe5219c9599661 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 330247b8434295c6a81a393b8408beaa parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c95ff939-d0c8-4e49-a36a-915253463efa_characterizeImage_1123957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c95ff939-d0c8-4e49-a36a-915253463efa_characterizeImage_1123957_58.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/342b720e-3ee4-470e-ab2a-dd2e2fd5a0fc_isr_1123957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/342b720e-3ee4-470e-ab2a-dd2e2fd5a0fc_isr_1123957_58.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4853f7a7321bfd105cfec27971ade546 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 927e529a27af16df85be97c9735f3127 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1f33aa00-e033-4b77-a719-79a04dac9b4e_characterizeImage_1123957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1f33aa00-e033-4b77-a719-79a04dac9b4e_characterizeImage_1123957_10.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1d7a51a4-eb7f-4d10-9282-a0c87bcb7133_isr_1123957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/1d7a51a4-eb7f-4d10-9282-a0c87bcb7133_isr_1123957_10.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98bfa41223aafa2715fdfade2f963fa8 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash f09886a46d581e9370265c46f1e5b8c1 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/ef56186b-3a83-4004-9477-722d911df0d4_isr_1124086_48.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/ef56186b-3a83-4004-9477-722d911df0d4_isr_1124086_48.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5372fc96c462c6ad92c44d4ff2db20ab with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash a9150f40e8bc9d82f6494b4660a393b5 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c2e0c1c5-8b7e-44d5-86f3-5dfdd437fd22_characterizeImage_1123957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c2e0c1c5-8b7e-44d5-86f3-5dfdd437fd22_characterizeImage_1123957_50.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cc6c25dc-c4f6-413d-b6c7-582f0133dfe0_isr_1123957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cc6c25dc-c4f6-413d-b6c7-582f0133dfe0_isr_1123957_50.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3edd5a6aca1e92354d3f76003c1599d with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 8afb08df356ec4084e39749eb4a1f3ad parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4aae4d14-bd14-459a-b465-8c5a484dc5ce_characterizeImage_1123957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4aae4d14-bd14-459a-b465-8c5a484dc5ce_characterizeImage_1123957_25.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4a280587-21b3-4279-917f-3baf566926e1_isr_1123957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4a280587-21b3-4279-917f-3baf566926e1_isr_1123957_25.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16bd79d2b09e0c28651395dfd4961d0 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash c975a55f428d35f779199589d5d1d111 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e4823db7-8356-486d-9ab0-0f57788687fb_characterizeImage_1123957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e4823db7-8356-486d-9ab0-0f57788687fb_characterizeImage_1123957_47.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9d5a2370-c560-4af4-9291-b53d22b4cae3_isr_1123957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9d5a2370-c560-4af4-9291-b53d22b4cae3_isr_1123957_47.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84a11c98041aa53739624f0780f379b1 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 24c76bf06b06d647d038149601c67b11 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/859c40dd-872c-4e72-a1a0-6d9f66058f2d_isr_1124086_20.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/859c40dd-872c-4e72-a1a0-6d9f66058f2d_isr_1124086_20.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdc6d5f858cc68d16a5ee60e80d0ed9e with result from task 216 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:41 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 0248b0e86b2c3ebed16bccece728f102 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c1cc1fcb-0aed-436c-9cf0-b217b12a241f_characterizeImage_1123957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c1cc1fcb-0aed-436c-9cf0-b217b12a241f_characterizeImage_1123957_13.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/81ee7c42-1484-4a85-b66e-16dbfac5f602_isr_1123957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/81ee7c42-1484-4a85-b66e-16dbfac5f602_isr_1123957_13.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea746956fd65b4837279ddfd094fa54b with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 84bfccfa0c8b87bdea82c39522e90c55 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639894e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fad9b6f1-0456-47e0-abf7-2ce7d885c528_characterizeImage_1123957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fad9b6f1-0456-47e0-abf7-2ce7d885c528_characterizeImage_1123957_40.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38dba4c1-05db-41c1-97c2-740e7d4b6c77_isr_1123957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38dba4c1-05db-41c1-97c2-740e7d4b6c77_isr_1123957_40.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d29585686f1ed6c1a8b5b416c7bee095 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 29f00f12b2f4a62826b89a1ca40aa4ab parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c64c503d-d91f-410c-9575-6ade0fd92dda_characterizeImage_1123955_6.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c64c503d-d91f-410c-9575-6ade0fd92dda_characterizeImage_1123955_6.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a083fa1d-072e-4031-9cf5-2ed0a250a0e4_isr_1123955_6.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/a083fa1d-072e-4031-9cf5-2ed0a250a0e4_isr_1123955_6.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 717b8e8d21a88feb83d2b9e53e72975b with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash a8f76c6f5fad4c8b246f9809be7f2be2 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1311e7a1-3496-4525-8b63-de772d437b74_characterizeImage_1123953_1.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1311e7a1-3496-4525-8b63-de772d437b74_characterizeImage_1123953_1.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e0b19ec2-e1f3-4e93-9f22-ef0215973612_isr_1123953_1.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e0b19ec2-e1f3-4e93-9f22-ef0215973612_isr_1123953_1.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd50f31429c330cfb8b0cdbf743c707 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 75ac6dac25655bb7b97dcc33c9ae43c3 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639622a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a73e210d-12a5-404f-a7dc-05b98dc9e9f1_characterizeImage_1123957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a73e210d-12a5-404f-a7dc-05b98dc9e9f1_characterizeImage_1123957_22.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e2cfd06f-a720-49c2-9a82-abb3cab17bbc_isr_1123957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e2cfd06f-a720-49c2-9a82-abb3cab17bbc_isr_1123957_22.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8adf3773e7fa3c5bfbfc39b8f94d456 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 1818d97f6f9ed034688676c0de8e94c6 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a1a02e5-451e-428b-817a-f6a82123ed9a_characterizeImage_1123957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a1a02e5-451e-428b-817a-f6a82123ed9a_characterizeImage_1123957_48.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ebdaf148-101c-45d1-b42a-b04f733c3377_isr_1123957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ebdaf148-101c-45d1-b42a-b04f733c3377_isr_1123957_48.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6660926b02e3422f49ea6d13bd5d92d with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 97ece9fc32bb40738a50be9516ad7430 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639add00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/47cbe57f-413a-4a65-8ad6-579c24a5301e_isr_1124086_32.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/47cbe57f-413a-4a65-8ad6-579c24a5301e_isr_1124086_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:41 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bb4953a96c90856b9b7edac48d76326 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 2d022c45144f0d07758316012d546f8a parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70ccac7a-126c-4146-8cc5-124fc02d3727_isr_1124086_49.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70ccac7a-126c-4146-8cc5-124fc02d3727_isr_1124086_49.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a5b935273362097175e005c335f4152 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash dd9d58b0a42d6eb58f8f214eafbfeac7 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639891c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b6930d35-0bda-4021-8054-f2522b156157_isr_1124086_6.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b6930d35-0bda-4021-8054-f2522b156157_isr_1124086_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:41 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46790ed57de18621f1579ca4b63bad17 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 8cb4dfd39931dfe77a8999685fbf97a8 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5a3e6023-215a-49ad-92d8-ee8c7495ad1f_characterizeImage_1123955_18.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5a3e6023-215a-49ad-92d8-ee8c7495ad1f_characterizeImage_1123955_18.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/96905294-674e-4ab8-b117-fcf4df557c83_isr_1123955_18.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/96905294-674e-4ab8-b117-fcf4df557c83_isr_1123955_18.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e60f4ee58a4a6c4a3d2edd8bc2fde081 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 0182ff3eeca0eb6ea62e74bade1ed6ab parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/06f4515a-6953-49b5-8680-5422e5c0dc8b_characterizeImage_1123955_52.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/06f4515a-6953-49b5-8680-5422e5c0dc8b_characterizeImage_1123955_52.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/411c5f6b-16c9-49f2-a24d-9ac0f502babb_isr_1123955_52.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/411c5f6b-16c9-49f2-a24d-9ac0f502babb_isr_1123955_52.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49e3a8af594917f26ac66ee0f51a5f1e with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash a4b5d9d321facba64ac4ef6faeb291ac parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3a86d2d8-b169-4c8c-a785-b21a47dbd2fc_characterizeImage_1123955_45.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3a86d2d8-b169-4c8c-a785-b21a47dbd2fc_characterizeImage_1123955_45.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/841f4046-60e9-46d7-8ced-d6d527cf5ce7_isr_1123955_45.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/841f4046-60e9-46d7-8ced-d6d527cf5ce7_isr_1123955_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:42 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61545ad3734b025b3f486a044566af65 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash bcd612b1806e3ab0b7a24345a12becdf parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ff6ba8b3-820f-4b18-b032-6b40c251ccec_characterizeImage_1123953_34.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ff6ba8b3-820f-4b18-b032-6b40c251ccec_characterizeImage_1123953_34.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c45bbd12-7e27-46dc-9b8e-494cf4af432b_isr_1123953_34.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c45bbd12-7e27-46dc-9b8e-494cf4af432b_isr_1123953_34.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 098bf7a1b42f9fb4c4c3b6701f88b6c7 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash c9af7d88fe2d1e9917c92faab15a58df parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4a21411e-2c3f-4396-af4e-167a3ee77838_characterizeImage_1123957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4a21411e-2c3f-4396-af4e-167a3ee77838_characterizeImage_1123957_4.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/8ae84bfe-9ece-44ee-ae34-0bf38934b29f_isr_1123957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/8ae84bfe-9ece-44ee-ae34-0bf38934b29f_isr_1123957_4.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5eba0a6bf6059200ef1e0f340353b3d3 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash a704ab4d1dd1738c9d5fd0e11ca02d64 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:42 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bc60964e-e2c5-4a03-a40a-7402b4eb21fa_isr_1124086_53.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/bc60964e-e2c5-4a03-a40a-7402b4eb21fa_isr_1124086_53.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bec6cc7a3ccdd1ecb73aa5c07edc6934 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash a34ef020eff8f6f050f0786277892043 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/26ef2bcf-60b9-4c72-8dad-89ab38146fc8_isr_1124086_36.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/26ef2bcf-60b9-4c72-8dad-89ab38146fc8_isr_1124086_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:42 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40c6e7ac23cfb904469a06518831c5ef with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 2b6281fdad5f4b39219f55c0350462fe parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c3484b53-588a-4972-b138-6d66e4e07437_characterizeImage_1123955_20.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c3484b53-588a-4972-b138-6d66e4e07437_characterizeImage_1123955_20.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5d9f7c82-8bfe-4d1b-ba68-1a2a09dfa2b5_isr_1123955_20.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5d9f7c82-8bfe-4d1b-ba68-1a2a09dfa2b5_isr_1123955_20.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 233952ebae6855f849d44208fe9f5206 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 3c76b1bfb79aac8150ec88ae347e1d79 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb7a67a4-f503-4240-b893-fedf5fa8e9b5_characterizeImage_1123953_29.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb7a67a4-f503-4240-b893-fedf5fa8e9b5_characterizeImage_1123953_29.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ce8b27c3-9e29-45be-a9e1-32da9819cc1d_isr_1123953_29.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ce8b27c3-9e29-45be-a9e1-32da9819cc1d_isr_1123953_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:42 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 733852fbf40834cb9b3a2ee9c35be767 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 7b6ff2c8ebaa5d2bb20c7fa2e2fb1a4b parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/607b01fd-d1d2-4843-989b-ffe9fced48dd_isr_1124086_38.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/607b01fd-d1d2-4843-989b-ffe9fced48dd_isr_1124086_38.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7ac1ed4a04f38c2a572c994f5a1866f with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash ee6a5e8a80bc064569e6f60791c72a01 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/666772c0-15df-4f20-8be1-2723440d7aab_isr_1124086_23.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/666772c0-15df-4f20-8be1-2723440d7aab_isr_1124086_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:42 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36aec3cb16b72f2eae497026a02a47af with result from task 246 parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:43 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 23c069ba06489649241b3161ff4213b1 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/11d5d40b-ede6-4162-a4e2-3839ef1e7798_isr_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/11d5d40b-ede6-4162-a4e2-3839ef1e7798_isr_1123955_55.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ad2813ebdadc22fd55aa6c38c97dabe with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash c3d49b98236bbd759e12c588adbade8e parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639625c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/43f8c892-8357-4e3e-ab2e-73b5a015101e_isr_1124086_27.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/43f8c892-8357-4e3e-ab2e-73b5a015101e_isr_1124086_27.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba8e7b158ea0fa2e71edb1a934481f1 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 520d9b2064cccd9ca720a96a980ca4c8 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70b437a1-2f85-4b38-8f31-9d6397edf1ad_isr_1124086_34.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/70b437a1-2f85-4b38-8f31-9d6397edf1ad_isr_1124086_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:43 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:43 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6307e08b1ea521635c13ee3aaadc260c with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash b1f3810d6e2c064aa39e39a410e7fe46 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639afc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/23be0b7c-9250-4197-b69e-f34362de0ef9_isr_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/23be0b7c-9250-4197-b69e-f34362de0ef9_isr_1123957_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:44 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54104cd9c42aa46e857015c90c08297e with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 2d08d6cf00659c1696c221fba9d0ce59 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e0619304-9f6f-44a4-ae05-0d549ec5844d_characterizeImage_1123953_51.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e0619304-9f6f-44a4-ae05-0d549ec5844d_characterizeImage_1123953_51.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e24fbfde-981e-4c81-97fd-6cc8a09b4fe6_isr_1123953_51.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e24fbfde-981e-4c81-97fd-6cc8a09b4fe6_isr_1123953_51.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2e8e82d88b0b52b2316bd62a44e29a6 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 0d2b40c17f5ab26685b9ad1441a8518b parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/20db20b6-a976-4252-b5e6-df5610c8998e_isr_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/20db20b6-a976-4252-b5e6-df5610c8998e_isr_1123957_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:44 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae54efc87d10c935bf7cbf87fd1681b5 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 22a6ec1a6950101db4b64c45b65820bf parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/2cef785f-d0f8-4973-a5dc-6cb0523b41a9_characterizeImage_1123957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/2cef785f-d0f8-4973-a5dc-6cb0523b41a9_characterizeImage_1123957_27.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/eb82beb1-20bf-420d-a3b1-d77b219f2b7f_isr_1123957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/eb82beb1-20bf-420d-a3b1-d77b219f2b7f_isr_1123957_27.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a023b2b8d4477b45e828e093993f46b5 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 4468ec30f1497c7fc5358bf13609d8ed parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/87e574c6-8b1b-4714-a6c1-67f1120d79c6_isr_1124086_12.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/87e574c6-8b1b-4714-a6c1-67f1120d79c6_isr_1124086_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:44 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:45 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:45 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:45 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13fde97b3fd8f2086ea0b2a5140f0f90 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 3894354370e87de3d121415413292e56 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/18fd6db9-6b6c-407d-9fc4-f37beb44c076_characterizeImage_1123953_7.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/18fd6db9-6b6c-407d-9fc4-f37beb44c076_characterizeImage_1123953_7.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea80c917-e965-4b79-a523-857785a75829_isr_1123953_7.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ea80c917-e965-4b79-a523-857785a75829_isr_1123953_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:46 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:46 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:46 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:47 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:47 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:47 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:47 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:47 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 772ad20c9445c77ac47e614a41e2f693 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash acf2b064af358e67396ef9f14da38cac parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/a23b154c-299b-44c3-81cd-d191de6055e8_isr_1124086_7.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/a23b154c-299b-44c3-81cd-d191de6055e8_isr_1124086_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:48 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20bf155f20e4293805ab3d0fdb263d36 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 707893ede8ac46c96281bb47d3b9d1d5 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a5779976-d425-4b1c-99fb-94ddf32b7047_characterizeImage_1123957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a5779976-d425-4b1c-99fb-94ddf32b7047_characterizeImage_1123957_18.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/d5e7aa8e-ca98-4738-97a8-941d4da52a38_isr_1123957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/d5e7aa8e-ca98-4738-97a8-941d4da52a38_isr_1123957_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:48 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:48 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:48 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:49 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:49 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:49 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:39:49 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e31453e923a44f6d34f753f29aee0ca with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 9d330e4c97e2b8d9486e21f7e23e1c68 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/290aa6c0-3be5-4c45-af83-1af64982d268_isr_1123953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/290aa6c0-3be5-4c45-af83-1af64982d268_isr_1123953_2.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40e781a16e76e73742e26b7c2ce4a5b9 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 23909fe7c63d1a66812555eb529fbcc4 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/66a4c7ba-1a05-47ef-81e8-fb4f93ece49c_isr_1124086_39.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/66a4c7ba-1a05-47ef-81e8-fb4f93ece49c_isr_1124086_39.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 073ad50616a1e643c90960b5730fe99a with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash a157bf2c93e31a3d1d0d71bab98c0079 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628084a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/360ed409-b291-4454-8c50-46802605073d_characterizeImage_1123957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/360ed409-b291-4454-8c50-46802605073d_characterizeImage_1123957_5.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba0a732e-3168-45e7-82fd-336de32b8d03_isr_1123957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/ba0a732e-3168-45e7-82fd-336de32b8d03_isr_1123957_5.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b76be64c8e9d40dc135357b3a1734a with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 919f2ac9cdda7f74305711c309b332ca parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/05b13db4-a38a-461f-b7dd-1e8eddd33a71_characterizeImage_1123957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/05b13db4-a38a-461f-b7dd-1e8eddd33a71_characterizeImage_1123957_32.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4ea86fab-3869-4eb8-a956-1709bfdccf0f_isr_1123957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/4ea86fab-3869-4eb8-a956-1709bfdccf0f_isr_1123957_32.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 249389e8177138f6d820fcbb98eb6664 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 8448a8bb24bfa6eeee15e183b2e321d0 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1e2c2e78-85e1-4789-a56a-c7cab4789362_characterizeImage_1123957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1e2c2e78-85e1-4789-a56a-c7cab4789362_characterizeImage_1123957_34.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cb2f9438-0615-4dd6-b4f4-8c0449a19db6_isr_1123957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/cb2f9438-0615-4dd6-b4f4-8c0449a19db6_isr_1123957_34.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63001321a3d37b87adcb85ed4e6f2bfc with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 01503387b5ab9b5094ae0c060565ca2c parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0d79ead7-d33d-4717-95a2-01cd7bbe9c92_characterizeImage_1123957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0d79ead7-d33d-4717-95a2-01cd7bbe9c92_characterizeImage_1123957_1.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/379603cf-b2d5-435b-8d98-447a3525296c_isr_1123957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/379603cf-b2d5-435b-8d98-447a3525296c_isr_1123957_1.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bdb1e4ce7a2c646f6c9bd8c9869c8fa with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 0a1aee431e08fa8cbe405271ac20575a parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a52f4ec6-7f6a-4952-a896-0fcb926bdd75_characterizeImage_1123953_41.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a52f4ec6-7f6a-4952-a896-0fcb926bdd75_characterizeImage_1123953_41.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/84d6257d-2500-426a-bfc5-ccb8c356511a_isr_1123953_41.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/84d6257d-2500-426a-bfc5-ccb8c356511a_isr_1123953_41.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e99c6551e33cd07199f51ca23a3a18e with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash aa335fa859d484b9cb909b927c97d81b parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f4dfc7a6-8392-42ee-aa36-7c6267d159ac_characterizeImage_1123957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f4dfc7a6-8392-42ee-aa36-7c6267d159ac_characterizeImage_1123957_56.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/3509bdd7-01eb-4220-9244-89d53b386399_isr_1123957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/3509bdd7-01eb-4220-9244-89d53b386399_isr_1123957_56.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b5e40e2dfd3011df9fa197eff3f4f38 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 13abd2443d18d526c6a0fa200294137d parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2f7f86d5-554e-43fe-94e8-5ddb7d6317c5_characterizeImage_1123953_35.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2f7f86d5-554e-43fe-94e8-5ddb7d6317c5_characterizeImage_1123953_35.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c2c583f8-9a1b-4c56-b37b-dd86e02d117c_isr_1123953_35.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c2c583f8-9a1b-4c56-b37b-dd86e02d117c_isr_1123953_35.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecbb8f9a8edc10635f038ff312ce2524 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 9e5de02eab2fe0f0792084814768a1b4 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/60a1afe4-8a2f-4b2f-ad78-6bbbdd50cdce_characterizeImage_1123957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/60a1afe4-8a2f-4b2f-ad78-6bbbdd50cdce_characterizeImage_1123957_35.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/97983f87-8eb6-46a2-8cd8-6cd56d84b0f3_isr_1123957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/97983f87-8eb6-46a2-8cd8-6cd56d84b0f3_isr_1123957_35.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d93fa6d9c9f388385631b6c990741e00 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 1c731bfbed0ebde45e6ab1f1e4995d36 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628087c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/37ba1208-01cb-429d-986f-1e22c1c0bf73_isr_1124086_46.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/37ba1208-01cb-429d-986f-1e22c1c0bf73_isr_1124086_46.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d914292e8c580de897872ada0e6db786 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash c28f4bf5b893737d3b073661120fb2ec parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c5cf59e9-8de7-416c-9b72-cf707c4d22ab_isr_1124086_25.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/c5cf59e9-8de7-416c-9b72-cf707c4d22ab_isr_1124086_25.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 793834a4097c7d935a532b36a94866d7 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash e6c8f7a701148714415480c763f89556 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/0f547878-8926-418f-a03b-629e5f0eb998_isr_1124086_11.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/0f547878-8926-418f-a03b-629e5f0eb998_isr_1124086_11.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3d0a9e4895d7938ca550fba376fd052 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash e68f2ee4405bcd41514d3417ddd162f5 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5400c5d-ad98-4948-8700-3008e99a1a0e_characterizeImage_1123955_62.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5400c5d-ad98-4948-8700-3008e99a1a0e_characterizeImage_1123955_62.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/3f397b55-1c40-4420-bafb-921a6a6883bd_isr_1123955_62.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/3f397b55-1c40-4420-bafb-921a6a6883bd_isr_1123955_62.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a77f7c3c8a4246cc53d2a558cc93abfc with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash a62a0ec46601102648cc5a82724b1af0 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/890e3c71-0cf9-40c8-afd1-b4ea2a827820_isr_1124086_4.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/890e3c71-0cf9-40c8-afd1-b4ea2a827820_isr_1124086_4.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 327dca35edffb1ca41b32a4b2956f395 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 56ace1f57ff683bfa0511c6d6b5e0dd4 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628994e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/5ac0beef-a1fc-40ff-87ce-216c4bc181b8_characterizeImage_1123953_26.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/5ac0beef-a1fc-40ff-87ce-216c4bc181b8_characterizeImage_1123953_26.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/de6a8285-abf7-4204-a3a6-70d81caf8de2_isr_1123953_26.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/de6a8285-abf7-4204-a3a6-70d81caf8de2_isr_1123953_26.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8204efca3be194a66feaa72de90102f with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 0279f28fc5518425b4b48e3de838209c parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ee763f54-4805-428c-a643-9e67752341b8_characterizeImage_1123955_35.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ee763f54-4805-428c-a643-9e67752341b8_characterizeImage_1123955_35.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d8054932-cfb3-479d-a53d-f0704c97c135_isr_1123955_35.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d8054932-cfb3-479d-a53d-f0704c97c135_isr_1123955_35.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4d627265b87f3baeba3b25a211a511d with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 87bee4c0de33398420f729eca878d502 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6f764238-b62c-461c-b22c-4d89de1f9ecf_characterizeImage_1123957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6f764238-b62c-461c-b22c-4d89de1f9ecf_characterizeImage_1123957_37.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9797d463-038d-4168-bc3b-603cecaba8db_isr_1123957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9797d463-038d-4168-bc3b-603cecaba8db_isr_1123957_37.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07aa4be9a99a31e1d3f7ae8a236a61c5 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash b47d990e69026e78dde2ce4f4274d2d6 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/68bcce6d-3b45-42b5-9cd7-3ecfae94d229_characterizeImage_1123955_57.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/68bcce6d-3b45-42b5-9cd7-3ecfae94d229_characterizeImage_1123955_57.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7fc2350c-90a8-4d08-ad56-00e133e59c2c_isr_1123955_57.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7fc2350c-90a8-4d08-ad56-00e133e59c2c_isr_1123955_57.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98349ee0bab947d98bf4d2ef7a42b082 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 110da3a3ae3d9fc14dbb7e0f0a8b0ab6 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cb7f842-19bd-45dd-80eb-4058cc3fef1d_isr_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9cb7f842-19bd-45dd-80eb-4058cc3fef1d_isr_1123955_26.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 667b9ac4f7c94a2bb63eb25644610761 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 1ed1e24022251cfbcf51f147c187a4cd parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a4fa284c-c76b-4cfa-a7c0-d50482dc5e8a_isr_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a4fa284c-c76b-4cfa-a7c0-d50482dc5e8a_isr_1123953_23.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e004e1c162064534001858811bff451d with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 1857f6e1e5a64c266aa2be81cf1c40eb parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a43f5656-7d28-41b2-9005-a9665a4d48b2_isr_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a43f5656-7d28-41b2-9005-a9665a4d48b2_isr_1123957_44.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca50f9fc868030cc04e234f1ffef2f2 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash f798b034d9f2b31e11ae711acfa73dcd parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a680af47-7bd4-4377-a9f8-a41f0eb4628b_isr_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/a680af47-7bd4-4377-a9f8-a41f0eb4628b_isr_1123953_43.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead77ab8788370c5017b25fcc0457d15 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 6474b7c372920189ee79de61d33b7648 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad18e30f-9692-4290-a19e-9163f802e1f4_isr_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ad18e30f-9692-4290-a19e-9163f802e1f4_isr_1123955_56.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9410a0ae5976706fec047e5a2e065f11 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 54b4fdbd1fe314ca6b36a95a66708743 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/96b7233c-b0b0-49a1-9261-c4af0d23916c_isr_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/96b7233c-b0b0-49a1-9261-c4af0d23916c_isr_1123953_18.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31bf25c0f3c153d9abdb4a44db2ec93b with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 1c20e1eead23c0304b3b5a9fdecf68b2 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38c97d87-4e9b-4f73-ad26-5316f7ae9133_isr_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/38c97d87-4e9b-4f73-ad26-5316f7ae9133_isr_1123957_59.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d51d2460cef4f4e22136ea2751009414 with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 46005d271901f66057883f74f687121f parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/e7b8f00b-5daf-4a19-8a89-ea9dcd20a589_isr_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/e7b8f00b-5daf-4a19-8a89-ea9dcd20a589_isr_1123955_4.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39526a58fa5d3bb35835efb980954141 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 56ed1e3c7d36d2aa67096258be0225e9 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/7145d158-c07e-4cac-a408-9060bff07cc5_characterizeImage_1123955_16.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/7145d158-c07e-4cac-a408-9060bff07cc5_characterizeImage_1123955_16.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f8cbbe59-acaf-4daa-9cbd-945740b790f7_isr_1123955_16.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/f8cbbe59-acaf-4daa-9cbd-945740b790f7_isr_1123955_16.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfafdcc26534e34bc30b4e4f4d71e67f with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash d67266c64715abf95f829cedfcff852b parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/12065c99-f0bc-4aab-b178-0b90a67a2e9a_characterizeImage_1123953_14.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/12065c99-f0bc-4aab-b178-0b90a67a2e9a_characterizeImage_1123953_14.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d1432bda-d40f-452c-9d2f-27968e7cbd12_isr_1123953_14.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d1432bda-d40f-452c-9d2f-27968e7cbd12_isr_1123953_14.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b43c566c4406a23733d07ddf19c829e5 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash ec781539d66d719ad40bb6e784d5019f parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d1037579-f28d-4b6e-9e90-b3a786a5e4ee_characterizeImage_1123955_29.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d1037579-f28d-4b6e-9e90-b3a786a5e4ee_characterizeImage_1123955_29.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/387492be-4140-472f-b816-18befdfc650f_isr_1123955_29.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/387492be-4140-472f-b816-18befdfc650f_isr_1123955_29.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 150438d03ff908d6d50007937fd0c94c with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 32e6715dcddae07fca8bfde23297880f parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8ca57102-0929-4172-aa6d-d74d2127065a_characterizeImage_1123955_30.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8ca57102-0929-4172-aa6d-d74d2127065a_characterizeImage_1123955_30.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/941afe33-bd9d-4d07-951c-686983ef1646_isr_1123955_30.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/941afe33-bd9d-4d07-951c-686983ef1646_isr_1123955_30.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 667f42530426091c098d21709eaef3ce with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 9203958242ace9238835f884f77597a6 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/681ae588-d7d7-4427-9016-88cdf6b26534_characterizeImage_1123953_32.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/681ae588-d7d7-4427-9016-88cdf6b26534_characterizeImage_1123953_32.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8b410eaf-2103-4a4c-a5bd-05b2d0b00432_isr_1123953_32.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8b410eaf-2103-4a4c-a5bd-05b2d0b00432_isr_1123953_32.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2fb7e572b322969ec75a539924d37e9 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 31b84a3b71164a0e0a55d387eb617bd3 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/42c5c225-eb89-4cd6-9bbc-f97f8693a56e_isr_1124086_41.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/42c5c225-eb89-4cd6-9bbc-f97f8693a56e_isr_1124086_41.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7328a30eb273ec07eeb23a5137c43366 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 62afcf3290f2c5b0c108296b415ac856 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a6040370-0f34-477e-ba11-ba02ed223e0f_characterizeImage_1123953_54.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a6040370-0f34-477e-ba11-ba02ed223e0f_characterizeImage_1123953_54.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/3a815154-b921-4035-8193-2d3db4614c8e_isr_1123953_54.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/3a815154-b921-4035-8193-2d3db4614c8e_isr_1123953_54.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7813a0500ea1916350f66408edf12608 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash adcc4104c627a086e93f8326e9d728f3 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/784be170-f29f-4282-8b76-0334bb12e0a5_characterizeImage_1123953_11.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/784be170-f29f-4282-8b76-0334bb12e0a5_characterizeImage_1123953_11.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f70ae8c4-240c-46c1-9cc9-3e7baab33064_isr_1123953_11.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f70ae8c4-240c-46c1-9cc9-3e7baab33064_isr_1123953_11.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c5adacf7b4ba5fc29b869d7dbe6bbc6 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 7e2788467f8d5fa379c288d9034741ae parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2e675c9c-76be-44a9-95f4-fcae68e9c68d_characterizeImage_1123955_21.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2e675c9c-76be-44a9-95f4-fcae68e9c68d_characterizeImage_1123955_21.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/b74f84e3-40c8-4f75-bef8-c78c383d4962_isr_1123955_21.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/b74f84e3-40c8-4f75-bef8-c78c383d4962_isr_1123955_21.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0aef39a94160888b5e32c2bc1ec539d with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash eac9364a230113d22ceff5dd36acd291 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628991c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9228ccce-a968-4936-933e-a3720b1e6c5c_isr_1124086_26.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/9228ccce-a968-4936-933e-a3720b1e6c5c_isr_1124086_26.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9840f1fac0ab8b8ce11529513d2e5cb9 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash dc6f9dbb7f1a31cbfe08969ac84f85d4 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3d8cd9eb-2773-4d42-945d-c8d4cc2f732b_isr_1124086_1.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3d8cd9eb-2773-4d42-945d-c8d4cc2f732b_isr_1124086_1.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f959bbbc1911812bb9587319511fb22 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 2c1c88107ce808a9d707329c2b4d7221 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2c1fdc04-c68d-4e20-b33e-f4c617451bf5_isr_1124086_21.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2c1fdc04-c68d-4e20-b33e-f4c617451bf5_isr_1124086_21.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d94f6bff79f8ae35068353af2839f3d with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 1f93203f4e9f414f54589472d4f8fad8 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2221bd5a-b167-4674-bfe9-c2ee492ec947_isr_1124086_14.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2221bd5a-b167-4674-bfe9-c2ee492ec947_isr_1124086_14.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 895155e120e14949284cc8409736aede with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 8f7ec507cde9853dd144343303d12d8c parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b7338235-7d56-4d18-a814-5336ba831353_isr_1124086_42.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/b7338235-7d56-4d18-a814-5336ba831353_isr_1124086_42.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 636c1a2c0846a955c0353e2c07a0b1a2 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash f25891e6c1f71b8133fa05832d433c7c parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/84dd5a21-e911-46ca-960b-ca3cce372eb8_characterizeImage_1123953_15.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/84dd5a21-e911-46ca-960b-ca3cce372eb8_characterizeImage_1123953_15.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ae7cfd0b-937c-495d-9142-89d6231ba2d3_isr_1123953_15.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ae7cfd0b-937c-495d-9142-89d6231ba2d3_isr_1123953_15.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 996859755a71862a8fa763cab54e6fe2 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 340ccf5ad7f7fcd63fbdcbe91c6297d6 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/685bd1c3-37b5-45ef-85bc-2f707b6bd544_characterizeImage_1123957_8.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/685bd1c3-37b5-45ef-85bc-2f707b6bd544_characterizeImage_1123957_8.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/31770d31-0203-4741-bef0-11c88dbb1b87_isr_1123957_8.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/31770d31-0203-4741-bef0-11c88dbb1b87_isr_1123957_8.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a6b5cb60e8e1a995677730c5821d7fb with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 5a82209d9a47057f17137adc82091cb4 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25388e84-da08-446f-b945-deb3cc070e76_characterizeImage_1123953_45.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25388e84-da08-446f-b945-deb3cc070e76_characterizeImage_1123953_45.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/69ed4e83-e716-4031-8b29-41ad78b1bb7f_isr_1123953_45.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/69ed4e83-e716-4031-8b29-41ad78b1bb7f_isr_1123953_45.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e762d5e51c1f989cf15607120ffa15ba with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 0c968a3db1002cd4529ece187dbae7f7 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/954084de-2208-46bc-a3e4-82b61949ba05_characterizeImage_1123957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/954084de-2208-46bc-a3e4-82b61949ba05_characterizeImage_1123957_14.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/09a6bfad-6116-424b-a06b-769040c6b2f4_isr_1123957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/09a6bfad-6116-424b-a06b-769040c6b2f4_isr_1123957_14.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bdd0168fab50fa9930d3ed77fa9186d with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash d60753152da78404cb8d5f35dbe2f2a1 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/14d8dbd9-2b70-436a-a66e-3efced8e02ef_isr_1124086_40.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/14d8dbd9-2b70-436a-a66e-3efced8e02ef_isr_1124086_40.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 488d5d074aa3212f7264659a07457be6 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash d6df22c1985fdff01a2d8c71f3363c5a parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/009e2acb-0e9d-4010-a696-07f57f8d8dad_isr_1124086_55.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/009e2acb-0e9d-4010-a696-07f57f8d8dad_isr_1124086_55.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bfb40e3c6f62eaa511be94657cd24e5 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 192c71489aa7635d1852ebd5671fc35e parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/85ad0103-6fa2-4b2a-82be-596344d3386a_characterizeImage_1123955_15.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/85ad0103-6fa2-4b2a-82be-596344d3386a_characterizeImage_1123955_15.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/10cf395f-3202-4ab2-a5c6-1e63b2be2fc2_isr_1123955_15.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/10cf395f-3202-4ab2-a5c6-1e63b2be2fc2_isr_1123955_15.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aff4d2e641e081561c04d4020a07f0e with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash b0668936f9e000eab8905cac2b346d94 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/659cb27b-1950-41df-aa27-82ec8c888d49_characterizeImage_1123953_16.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/659cb27b-1950-41df-aa27-82ec8c888d49_characterizeImage_1123953_16.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c31eb41e-8584-44d5-aa40-4f5d5d3a46a4_isr_1123953_16.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/c31eb41e-8584-44d5-aa40-4f5d5d3a46a4_isr_1123953_16.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75472cd05bfe37b8c427c304380cc84e with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash ad886ae1ef6ef69a2704742d185ddf87 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1450c694-0d8b-49ec-895d-199e43554d2d_characterizeImage_1123953_20.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1450c694-0d8b-49ec-895d-199e43554d2d_characterizeImage_1123953_20.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8188f60b-b563-40be-b63a-c17cd06717bf_isr_1123953_20.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/8188f60b-b563-40be-b63a-c17cd06717bf_isr_1123953_20.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83c1348ef45217264d1f3e818f0a105f with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash b44cb50f33a067e40b5b8fd97ea7fdf3 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/91abe08d-c6aa-47fa-b7fe-839aaddb7be5_characterizeImage_1123953_52.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/91abe08d-c6aa-47fa-b7fe-839aaddb7be5_characterizeImage_1123953_52.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6f5af050-e260-4757-abd8-16a063141a25_isr_1123953_52.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6f5af050-e260-4757-abd8-16a063141a25_isr_1123953_52.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0cd5b28acb3bc012bd735ce1da356f7 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 7ec436fb57766a003b66bd954cc0aadc parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f57f91fe-8616-454f-acdf-a11237775f68_characterizeImage_1123955_8.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f57f91fe-8616-454f-acdf-a11237775f68_characterizeImage_1123955_8.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8378df0c-1c62-4601-8e80-86600ea52d35_isr_1123955_8.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8378df0c-1c62-4601-8e80-86600ea52d35_isr_1123955_8.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64c8237b2c01f7d473480ae6c843cc7d with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash dc6e970bd064e1c3446d207b204bbd81 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7f66162a-621b-4983-b7f2-7c87c3675807_characterizeImage_1123953_48.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7f66162a-621b-4983-b7f2-7c87c3675807_characterizeImage_1123953_48.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ee4874c2-3991-4c9f-bdc4-5c3f6304dfd1_isr_1123953_48.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/ee4874c2-3991-4c9f-bdc4-5c3f6304dfd1_isr_1123953_48.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 500ce755ff6562b1ca7bc0a3c35ae73f with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash d1b417b96d2882b8c18fde2621871dcd parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5d6392f2-2dc2-42c1-8395-d98f8c69c778_characterizeImage_1123955_24.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5d6392f2-2dc2-42c1-8395-d98f8c69c778_characterizeImage_1123955_24.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7f5a96a3-fb5a-4cb5-87a1-672bf01307d2_isr_1123955_24.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/7f5a96a3-fb5a-4cb5-87a1-672bf01307d2_isr_1123955_24.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75202b5e6be856b162823242515b7918 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 0296ffc2b7f5ffdc2da21562474049a5 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/8fb0621b-fe9c-40e5-885d-de2f13bf2573_characterizeImage_1123957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/8fb0621b-fe9c-40e5-885d-de2f13bf2573_characterizeImage_1123957_12.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b57ced11-4505-45c7-8e45-62cec4afa864_isr_1123957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/b57ced11-4505-45c7-8e45-62cec4afa864_isr_1123957_12.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b446a2362f3e88371d507c0fc141a7 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 29ed57fc1f01d9bb43511ecbf6301c8e parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ed676976-b865-4896-8e93-d90d916fac31_characterizeImage_1123957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ed676976-b865-4896-8e93-d90d916fac31_characterizeImage_1123957_11.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/bdc10251-bc2b-4aa2-b3cf-9612fb8ba9a8_isr_1123957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/bdc10251-bc2b-4aa2-b3cf-9612fb8ba9a8_isr_1123957_11.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 324278c33a695fc2dc0fe5b70748915c with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash a9d1832a360c9a2c6233bb2591c7c2cb parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/680f9c63-1265-46b3-8ef4-fc7265724060_isr_1124086_52.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/680f9c63-1265-46b3-8ef4-fc7265724060_isr_1124086_52.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aad89865279dc4a0974f1310d383ea3 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash d3e8d57d82f9f29347c1357640fc354b parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bfb0fb72-e688-4652-8ec2-fbb281c6cfa9_characterizeImage_1123953_55.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bfb0fb72-e688-4652-8ec2-fbb281c6cfa9_characterizeImage_1123953_55.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/65601d88-6aac-4815-97b2-1d7d9df2447c_isr_1123953_55.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/65601d88-6aac-4815-97b2-1d7d9df2447c_isr_1123953_55.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf6005bcd28aaccbb24c98d61542f97 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 153d2a1537d55c51f0e27ecfd575f771 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/669aba42-d642-4505-86de-d33ba45963d4_characterizeImage_1123957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/669aba42-d642-4505-86de-d33ba45963d4_characterizeImage_1123957_36.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9f66d2e6-3653-40d0-bd10-9bd223476640_isr_1123957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9f66d2e6-3653-40d0-bd10-9bd223476640_isr_1123957_36.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4009486226c348233250041233e67a6a with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 04698d798d4bd343184502558f343f84 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/d6bf5932-9a26-4471-aad7-9e05c405d9e1_isr_1124086_13.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/d6bf5932-9a26-4471-aad7-9e05c405d9e1_isr_1124086_13.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dba5b71664f35741d4b1e3e998c474c8 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash b8a56a7bf7e517fd386f8e12df72befd parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/5e87df75-122e-47b4-aa07-f01f935f8371_characterizeImage_1123957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/5e87df75-122e-47b4-aa07-f01f935f8371_characterizeImage_1123957_17.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/281d078c-38fc-41f3-8e5e-11d6a8ec75f4_isr_1123957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/281d078c-38fc-41f3-8e5e-11d6a8ec75f4_isr_1123957_17.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d94896f515a1d3c9b6fd188260302be with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 5b0bb9accdf55c033e400bbe35f9ee4f parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1db1da34-bf7e-453e-9a0d-5a64837b29de_characterizeImage_1123957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1db1da34-bf7e-453e-9a0d-5a64837b29de_characterizeImage_1123957_55.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/58272c75-337d-4eb7-8a6c-5ac8149f8918_isr_1123957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/58272c75-337d-4eb7-8a6c-5ac8149f8918_isr_1123957_55.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1929f1223a701cf75c008505e804a7c with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash d8f6ce2152f5945106dc31ac06385d57 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/666722eb-57ff-48ad-bb6b-e65b99d222ca_characterizeImage_1123953_50.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/666722eb-57ff-48ad-bb6b-e65b99d222ca_characterizeImage_1123953_50.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e01b83df-d980-4030-8544-0f0525c89ba2_isr_1123953_50.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/e01b83df-d980-4030-8544-0f0525c89ba2_isr_1123953_50.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ddadf38f495373fbfdec25b3631961e with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash bd5a631cd809e1fd58d625f361886fe2 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/0bdcfb24-312a-4b8d-ae6f-70e5f9ed210a_characterizeImage_1123955_7.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/0bdcfb24-312a-4b8d-ae6f-70e5f9ed210a_characterizeImage_1123955_7.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2f567505-4b0c-49a7-a327-45355f0140bb_isr_1123955_7.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2f567505-4b0c-49a7-a327-45355f0140bb_isr_1123955_7.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e79180e648d8ab3d61c500c0c99539fc with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash b805c8d68401ae47a999552d269c681b parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/9ef4c472-4628-4b75-94fe-f43bd7895c0f_characterizeImage_1123955_46.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/9ef4c472-4628-4b75-94fe-f43bd7895c0f_characterizeImage_1123955_46.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2dbd943e-d115-41b3-acff-854908ba1f1d_isr_1123955_46.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/2dbd943e-d115-41b3-acff-854908ba1f1d_isr_1123955_46.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9aa0385f609f3e200ac3e0ca26affc3 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash f7af245d36596ef4c4f9435c61894e2c parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628982c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/883d04e5-f562-46ec-b6bd-976bc99c169c_isr_1124086_37.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/883d04e5-f562-46ec-b6bd-976bc99c169c_isr_1124086_37.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe85fe44e1be0a7b4fed8100f0640430 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 17e8ea096b6243368ca72ba73d85ae85 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a317be8-6552-44b8-8422-994cb2e9349c_characterizeImage_1123957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a317be8-6552-44b8-8422-994cb2e9349c_characterizeImage_1123957_26.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a0561586-68b0-47be-9c56-2440a2eb27d3_isr_1123957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a0561586-68b0-47be-9c56-2440a2eb27d3_isr_1123957_26.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea224b5541043477b2d1bd1766669d4 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash c132fc371ea34e953ba2b64ee73c4ea1 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/86743632-7506-4d83-87b6-ca56508ecaca_characterizeImage_1123957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/86743632-7506-4d83-87b6-ca56508ecaca_characterizeImage_1123957_9.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/72698028-6738-46af-bf17-d119aaf32000_isr_1123957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/72698028-6738-46af-bf17-d119aaf32000_isr_1123957_9.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b6d7befc694e845efba79bbb4b880a7 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash b960543338037143fd8119ca3cf52c02 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e24fdb1f-87cd-4d5b-a76e-e6350d507cd0_characterizeImage_1123953_38.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e24fdb1f-87cd-4d5b-a76e-e6350d507cd0_characterizeImage_1123953_38.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/544400de-f09b-4d11-9577-50de62023560_isr_1123953_38.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/544400de-f09b-4d11-9577-50de62023560_isr_1123953_38.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb671a63c678c6d89cfff2da9a48f890 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 454abd4bc0d448a57cdb310375233ce8 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/caf7ccbf-adac-4817-8d30-92147d85cd0b_isr_1124086_29.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/caf7ccbf-adac-4817-8d30-92147d85cd0b_isr_1124086_29.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee1ec5622fc264d05e5ccaa50ad370d with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash c3791ad784fa82a0f48917c19df80976 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef7d7d38-885e-45d6-98df-2d10fc25c800_isr_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/ef7d7d38-885e-45d6-98df-2d10fc25c800_isr_1123955_33.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54009002c2e58b1e47a23e1bc67ae82e with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 7548cefae1f1649d3075d0a093f2fc46 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bce4d9-0185-4ef7-8f0d-309f1f63f081_characterizeImage_1123955_1.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bce4d9-0185-4ef7-8f0d-309f1f63f081_characterizeImage_1123955_1.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/14070f88-bc91-412d-bfa2-e63b93c3b52b_isr_1123955_1.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/14070f88-bc91-412d-bfa2-e63b93c3b52b_isr_1123955_1.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 331ff92742e2f2a46f93b5a1681255e8 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 3f1d18262ef1c5363773431dd52f2d6d parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b4dd492b-3090-495d-b565-3645fd9f6dc8_characterizeImage_1123953_10.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b4dd492b-3090-495d-b565-3645fd9f6dc8_characterizeImage_1123953_10.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d5fce20c-ca80-468e-a149-60e026aea15c_isr_1123953_10.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d5fce20c-ca80-468e-a149-60e026aea15c_isr_1123953_10.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c7385854a5cb719a9e5acdf61012fae with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 479540b13b25978c268b7d83c7a3ee4d parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ebab0828-1e5c-4902-8a18-6bf52c160cf9_characterizeImage_1123955_58.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ebab0828-1e5c-4902-8a18-6bf52c160cf9_characterizeImage_1123955_58.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6b7bfe1f-1378-44aa-a936-ddf18acb888e_isr_1123955_58.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/6b7bfe1f-1378-44aa-a936-ddf18acb888e_isr_1123955_58.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c6a6793656dadf698d5299d256c4230 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 2bf0cbd27c0283b36a0dad924269a652 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9661cd3d-ed1e-4399-9806-ddbaca944a18_isr_1123957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9661cd3d-ed1e-4399-9806-ddbaca944a18_isr_1123957_2.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ace85c62c89b413c8cb82ee9270149dc with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash d064cea2d0ba266db62ec8663b27b261 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e5048c9b-61f1-4688-83aa-b49f93c3a403_characterizeImage_1123953_25.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e5048c9b-61f1-4688-83aa-b49f93c3a403_characterizeImage_1123953_25.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/cf850369-99f4-4783-995e-b9dda508ba53_isr_1123953_25.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/cf850369-99f4-4783-995e-b9dda508ba53_isr_1123953_25.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bce1bbf7b684948c9b20a7fedf983fb with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash bd20a3c757288cac0aab6495a16f2a4b parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/e3b2bbfd-43cc-4075-a6fb-3ea1928172d5_isr_1124086_60.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/e3b2bbfd-43cc-4075-a6fb-3ea1928172d5_isr_1124086_60.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1dc251550528fed8648c334c5721fe3 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 4b0fb9474321991e8acbdb77e04dd5de parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/704f4306-e8f6-464f-a99a-c4698fa513ba_isr_1124086_56.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/704f4306-e8f6-464f-a99a-c4698fa513ba_isr_1124086_56.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b8cd8ef30d3904b4709cddbf0cfd63a with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 603af55d92094939c23eac6d006feeb3 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e0005ede-a607-44b3-800b-31d0a0f29a70_characterizeImage_1123957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e0005ede-a607-44b3-800b-31d0a0f29a70_characterizeImage_1123957_49.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/af715220-2a51-4fb0-92f7-818e6764c157_isr_1123957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/af715220-2a51-4fb0-92f7-818e6764c157_isr_1123957_49.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34bae94a2ff3c90e40597de7804f58cc with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 0f90651eb258afdb3d48d4b209f202d2 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/6ab8b6e9-fe2d-4d34-82ea-6a3148074ba1_isr_1124086_59.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/6ab8b6e9-fe2d-4d34-82ea-6a3148074ba1_isr_1124086_59.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e8508e4a661a6fed0331a60304f096b with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash a6034ae8e3b23f47abdc90cc9d93c832 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/121b615d-324f-42f2-947a-692a2e88571c_isr_1124086_33.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/121b615d-324f-42f2-947a-692a2e88571c_isr_1124086_33.stderr 2024-03-07 00:40:05 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:40:05 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd2ab78c2d679fbd758fb659b0758ff with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 86f043d6c9387ab64c67a62550153e19 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/88d9ec60-5358-44db-a4ae-42842adef80e_isr_1124086_2.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/88d9ec60-5358-44db-a4ae-42842adef80e_isr_1124086_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:05 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:06 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0391e74a134eb15faebbb4cb33fa311c with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 9e05f1f0e7011d444b14311e0baa8b73 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/7c0055a0-3556-4dc4-86df-c9ea3725dffc_characterizeImage_1123957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/7c0055a0-3556-4dc4-86df-c9ea3725dffc_characterizeImage_1123957_46.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5335296d-dc14-4e37-878a-c851cccfd53e_isr_1123957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/5335296d-dc14-4e37-878a-c851cccfd53e_isr_1123957_46.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d9a16e788274a719c0baa0869bd5c83 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash b6109bfb4f8761b98c3ca8026e5682cf parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627298a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d23eaed7-dc68-4e77-aa71-6ab543edea65_characterizeImage_1123955_28.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d23eaed7-dc68-4e77-aa71-6ab543edea65_characterizeImage_1123955_28.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/c692388d-180d-46a2-8514-4c50f1cb5215_isr_1123955_28.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/c692388d-180d-46a2-8514-4c50f1cb5215_isr_1123955_28.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328bd7d49848b354a06d5e4cdcf06936 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 212953490d0acd361d42e62acbecfe7e parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/60638750-74ed-4a2f-8759-e70349dae14b_characterizeImage_1123955_51.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/60638750-74ed-4a2f-8759-e70349dae14b_characterizeImage_1123955_51.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8a892656-263f-4934-b3f3-76ae0600cd86_isr_1123955_51.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/8a892656-263f-4934-b3f3-76ae0600cd86_isr_1123955_51.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 767ad2b70e8f33964cc4fb32f5ea1425 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 62518c8975a173d1a57e5adf474c5737 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/cfc0c300-964f-45bd-adfe-aa0281bedd47_characterizeImage_1123955_49.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/cfc0c300-964f-45bd-adfe-aa0281bedd47_characterizeImage_1123955_49.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d0f186d6-3e76-4135-aef4-436e56be5d51_isr_1123955_49.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d0f186d6-3e76-4135-aef4-436e56be5d51_isr_1123955_49.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8424ee4c05bf2ac4c40e3ae8ca3154a with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 0367e1f834e61815d78aeb1d4ea1a9fd parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627285e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e7f574c-71e9-497b-a5d2-c8442d175e75_characterizeImage_1123955_50.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e7f574c-71e9-497b-a5d2-c8442d175e75_characterizeImage_1123955_50.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5a4f021c-1bca-4014-a083-8ea09cf1de8a_isr_1123955_50.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/5a4f021c-1bca-4014-a083-8ea09cf1de8a_isr_1123955_50.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b4959b2fa84cdc803f631e15dd2efe4 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 652d169eb91c0347f0abcc445b794f6e parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35d847b0-4f4e-439c-b57b-171f7814a66c_characterizeImage_1123955_38.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35d847b0-4f4e-439c-b57b-171f7814a66c_characterizeImage_1123955_38.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/53af09b6-5795-4f5c-b5a6-3fca45dc3c10_isr_1123955_38.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/53af09b6-5795-4f5c-b5a6-3fca45dc3c10_isr_1123955_38.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e38b223df25f3fb771ba65705d659cc with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 33dc19a5775839be1fd59fe2abec4431 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5a7625c-7036-49b7-a9cd-5812839634a6_characterizeImage_1123955_44.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5a7625c-7036-49b7-a9cd-5812839634a6_characterizeImage_1123955_44.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/1732f656-392d-4aa5-a73d-f96a002c51f1_isr_1123955_44.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/1732f656-392d-4aa5-a73d-f96a002c51f1_isr_1123955_44.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97e39788863405ff0663ee8efaaabd31 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash d7acc791ed6d365c80b4ff45f278be3e parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7d970114-a16b-46b1-9172-200e678a0f10_characterizeImage_1123953_27.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7d970114-a16b-46b1-9172-200e678a0f10_characterizeImage_1123953_27.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fab45c31-b313-4dcb-932a-01730d15c0cf_isr_1123953_27.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fab45c31-b313-4dcb-932a-01730d15c0cf_isr_1123953_27.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03dc67f7577570bb7360329764331ec1 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 418ffff0d25a5edb91bacf754d54d918 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4abfa646-3475-461c-bbb9-a2ca9036eb33_characterizeImage_1123957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4abfa646-3475-461c-bbb9-a2ca9036eb33_characterizeImage_1123957_57.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e955a409-1e33-4d48-a714-9cc5d6612d20_isr_1123957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/e955a409-1e33-4d48-a714-9cc5d6612d20_isr_1123957_57.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 813d414cb9e6d1ec34afc11dcff6cb01 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash bf7a06ed9eb0a5b3afd1cd25772b0434 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/1f544a5e-e822-4091-8e27-d3d476afda8f_isr_1124086_3.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/1f544a5e-e822-4091-8e27-d3d476afda8f_isr_1124086_3.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab768fc6e796c8676a55c0fd53f6e55e with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 1eb5af4c7b82d754d1de710d683f7fc7 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3c3eced2-c40b-465b-aa93-3107ecfe5c8c_isr_1124086_17.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/3c3eced2-c40b-465b-aa93-3107ecfe5c8c_isr_1124086_17.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fef9d2c09a12f9b725fad47b1b01331e with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash adcf0eed01884b39de975fd14e2d8cf8 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2924da02-44de-40b1-8417-4a20bb556815_characterizeImage_1123953_17.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2924da02-44de-40b1-8417-4a20bb556815_characterizeImage_1123953_17.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fa66b048-6f2d-4101-b790-67318fa3f740_isr_1123953_17.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/fa66b048-6f2d-4101-b790-67318fa3f740_isr_1123953_17.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80b46a2870b59f790522e881e24fe5b2 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 10934ebb98cd53f6a3ad7ba96900083d parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f761bebd-a901-4b5f-b7d5-283fa8547b44_characterizeImage_1123957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f761bebd-a901-4b5f-b7d5-283fa8547b44_characterizeImage_1123957_39.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/2c79cc35-7f20-446a-a7a2-fee560b9b411_isr_1123957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/2c79cc35-7f20-446a-a7a2-fee560b9b411_isr_1123957_39.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 493440d41b7126338978cab869353fb5 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash f5a521a388f9323e8a697c531976d2ca parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cfd24f69-0cb2-4250-9828-430c0006dfe5_characterizeImage_1123953_49.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cfd24f69-0cb2-4250-9828-430c0006dfe5_characterizeImage_1123953_49.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/03919008-a15d-44d9-a868-896712269880_isr_1123953_49.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/03919008-a15d-44d9-a868-896712269880_isr_1123953_49.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a5c4459cdb8643afba989922da5f0a6 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash be24cf2f69351ea2ea79d0cb66c19d43 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/7d269cd7-5313-49f0-bfcc-e5a3dddfffd6_isr_1124086_61.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/7d269cd7-5313-49f0-bfcc-e5a3dddfffd6_isr_1124086_61.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58529606e9d4f20b7131a577a72a06fb with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 962d60cd93b253fee1f388eb851d2f5b parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25382b56-5b0f-439f-9721-3ec5b044bede_characterizeImage_1123953_21.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25382b56-5b0f-439f-9721-3ec5b044bede_characterizeImage_1123953_21.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0dbf2003-a651-4dc8-8162-fdfd2af5a179_isr_1123953_21.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0dbf2003-a651-4dc8-8162-fdfd2af5a179_isr_1123953_21.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bc546f32e87aa0328502fe0d8e4f8db with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 39d3b26405abc4ca7beb981296397818 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e3921330-5f26-4366-83aa-aecc1b9a2d03_characterizeImage_1123957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e3921330-5f26-4366-83aa-aecc1b9a2d03_characterizeImage_1123957_21.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a63842bd-048c-489f-869c-897c2a25aa2a_isr_1123957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a63842bd-048c-489f-869c-897c2a25aa2a_isr_1123957_21.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28ba5087eef6b5b983c8209a6a09a28b with result from task 729 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:08 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 09f0b345cdc0d5812728b5c74eb0fb48 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/99e121d5-40dc-4659-a2d9-39825c48b34a_characterizeImage_1123953_19.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/99e121d5-40dc-4659-a2d9-39825c48b34a_characterizeImage_1123953_19.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/12b52545-5c67-4a72-9128-5aaba533f450_isr_1123953_19.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/12b52545-5c67-4a72-9128-5aaba533f450_isr_1123953_19.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b997fbfa73235567479b59344b8c4e5b with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash dba6ce411e50df38730eb9676b98d70f parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/3e3b12ea-f2df-476f-beee-2e237fb07aab_characterizeImage_1123957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/3e3b12ea-f2df-476f-beee-2e237fb07aab_characterizeImage_1123957_60.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a93e2329-5921-4b79-9256-1797ef369405_isr_1123957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/a93e2329-5921-4b79-9256-1797ef369405_isr_1123957_60.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4a659302b774d4120b08792edc95953 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 2380e60c745ae607acf1ba38f5be5cdb parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d0bd9e0-e7d7-4796-bc79-0aae515a8b19_isr_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d0bd9e0-e7d7-4796-bc79-0aae515a8b19_isr_1123953_5.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ae752aa92a117bc5f845923b027521d with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash f5730bf1c137ef641a399d64c0016e00 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ea8ca52c-7b9e-4b81-9cb8-a0a2436a2847_characterizeImage_1123957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ea8ca52c-7b9e-4b81-9cb8-a0a2436a2847_characterizeImage_1123957_23.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/280ec7f9-1d83-4498-b281-4f3ee6fe92e0_isr_1123957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/280ec7f9-1d83-4498-b281-4f3ee6fe92e0_isr_1123957_23.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 395d876701c5225dbd56dd4d20cca876 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 27c7d93c049e3f6b3ea76e1c240039d3 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/90fcaa99-027a-49f5-80f1-f92257eedf28_characterizeImage_1123953_58.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/90fcaa99-027a-49f5-80f1-f92257eedf28_characterizeImage_1123953_58.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/59274085-eaa3-42db-9dec-4a8d5f4c1097_isr_1123953_58.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/59274085-eaa3-42db-9dec-4a8d5f4c1097_isr_1123953_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:08 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0055c57241f860768cd58d8ac138ce3e with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash ce256f55a742c3297803833183a3324a parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ac4f8612-829c-4958-8f4c-9417a421f66a_characterizeImage_1123957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ac4f8612-829c-4958-8f4c-9417a421f66a_characterizeImage_1123957_30.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9ffd4de9-36b4-41b0-9e65-ff61a24af9f1_isr_1123957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/9ffd4de9-36b4-41b0-9e65-ff61a24af9f1_isr_1123957_30.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4f45a139af7d8b446ae14a8caff9772 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 4b70da54bcbe1719e2a00c3a3b528b58 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/324cf0a0-f1c6-40dd-aba9-328d48ef61d8_characterizeImage_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/324cf0a0-f1c6-40dd-aba9-328d48ef61d8_characterizeImage_1123955_31.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/00768c35-8008-44c2-8e15-5450cee27165_isr_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/00768c35-8008-44c2-8e15-5450cee27165_isr_1123955_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:09 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c79daf0860d8930d6c87e3d1ad3d3d with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 6bc6b94b2f9f4514773c6f5a48465cc5 parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cdddb0d9-f32b-4312-b1e9-72fd2fb9e02a_characterizeImage_1123953_39.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cdddb0d9-f32b-4312-b1e9-72fd2fb9e02a_characterizeImage_1123953_39.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d9add1a-0f63-47bb-8642-cfda683b11b4_isr_1123953_39.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6d9add1a-0f63-47bb-8642-cfda683b11b4_isr_1123953_39.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da002e15978fa415591432ae3768f80d with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 1efec5af9dd411c496b315d62191b645 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90524e26-ba6b-4ca0-b365-9dc42ec28545_characterizeImage_1123957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90524e26-ba6b-4ca0-b365-9dc42ec28545_characterizeImage_1123957_6.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/961d67d4-255c-438f-b86d-13171a2a2b09_isr_1123957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/961d67d4-255c-438f-b86d-13171a2a2b09_isr_1123957_6.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d789422a1ed5c18f3f2a3396f3158b25 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 54c0a4e6f31cd52c053b2731c5493b68 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f87aea17-d25a-4a79-aa8a-2fb04364237c_isr_1124086_22.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f87aea17-d25a-4a79-aa8a-2fb04364237c_isr_1124086_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:09 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6835db9d7fde96d209dfbf60deef06c with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 08e6f19561a85b5b6a68f608b6f6d653 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3ddc56b0-ac20-407a-89bf-c4d5649a9687_characterizeImage_1123953_60.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3ddc56b0-ac20-407a-89bf-c4d5649a9687_characterizeImage_1123953_60.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f8e4278e-9ecf-426b-8ab0-1eda9779cc20_isr_1123953_60.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/f8e4278e-9ecf-426b-8ab0-1eda9779cc20_isr_1123953_60.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 0 failed parsl.dataflow.dflow INFO: Task 4 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 44338a6df7e2e5c5fe5e16be33a4e281 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 1 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b58bee2d0b409450307113441c6ca72 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 4b2c0b62908ef6c5e468082bb994e87b parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/357e4691-9037-4188-b01f-13136d918359_isr_1124086_31.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/357e4691-9037-4188-b01f-13136d918359_isr_1124086_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:09 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:10 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc19af0e22105d0e46f60ff1ea0af167 with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 2d83904d48596e4f1ce4e3c62f762dac parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e8c35d5a-89f4-4677-b353-6d37bb881326_characterizeImage_1123955_61.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e8c35d5a-89f4-4677-b353-6d37bb881326_characterizeImage_1123955_61.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9bcde1f6-548a-4828-9f33-0a7760fd767c_isr_1123955_61.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/9bcde1f6-548a-4828-9f33-0a7760fd767c_isr_1123955_61.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b48630f31d664bade3cb1e754f815113 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash b620463452968fedcfc6f95f8ee88e0f parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bc04d8dd-d713-4671-93fd-2b64688d4a53_characterizeImage_1123953_12.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bc04d8dd-d713-4671-93fd-2b64688d4a53_characterizeImage_1123953_12.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/53cc511b-8f28-424d-a856-6cbb8b012cbe_isr_1123953_12.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/53cc511b-8f28-424d-a856-6cbb8b012cbe_isr_1123953_12.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9939964aeb31fd592621fda66d108bb with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 48b5a3340b4cbb113a64d14de752a40b parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/887a47fe-fcef-4446-b34c-58507494a503_isr_1124086_35.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/887a47fe-fcef-4446-b34c-58507494a503_isr_1124086_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:10 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:11 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:11 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:11 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:12 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 286a97a9cc6a09ee4cba62a0f18cc94e with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 8a79d7bf52e2cbb96762550b3eb969f7 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/252fc933-e65c-47d7-b856-a74f2bf442e5_isr_1124086_18.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/252fc933-e65c-47d7-b856-a74f2bf442e5_isr_1124086_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:12 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 73 try 0 failed parsl.dataflow.dflow INFO: Task 73 marked for retry parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 804ae82d5ae294d546a262a0813a41fa parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 1 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:12 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:13 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b11982a23d81dc8860accc296694ee7c with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 3ae032be8cd92868527afb26acdf35ec parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5035db55-a074-4f01-a75c-8ad541167f39_isr_1124086_45.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/5035db55-a074-4f01-a75c-8ad541167f39_isr_1124086_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:13 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:13 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:13 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:14 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 199 try 0 failed parsl.dataflow.dflow INFO: Task 199 marked for retry parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash f2d6e8b33b62550cd6ca9feefa25bf06 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 1 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:14 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:14 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 289 try 0 failed parsl.dataflow.dflow INFO: Task 289 marked for retry parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 57204c40005aedc45d3730f350fd0a55 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 1 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:15 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:15 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:16 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:16 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:16 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:17 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2ff700ec131fd9af7eef1d7acb91f61 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash f013f847bf0b8125e7cb1d403b2259a3 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2d30c17d-55a7-46cd-adda-bbdfd74c55da_characterizeImage_1123955_34.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2d30c17d-55a7-46cd-adda-bbdfd74c55da_characterizeImage_1123955_34.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d3c2c103-4af3-4d7d-9ebb-12a72d6503dc_isr_1123955_34.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/d3c2c103-4af3-4d7d-9ebb-12a72d6503dc_isr_1123955_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 0 failed parsl.dataflow.dflow INFO: Task 100 marked for retry parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 1f7b5a60331c086cf06ceed9750c6504 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 1 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:17 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:17 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:18 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f6b0632ea31581f7c98971490b54fc4 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 346fe0a28fabf318c4e598f6cfe58e23 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/646e2e0b-88cc-4844-a1a6-5aadc352a40f_characterizeImage_1123955_3.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/646e2e0b-88cc-4844-a1a6-5aadc352a40f_characterizeImage_1123955_3.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/36cd1b55-c9aa-4023-9b06-15080453aed1_isr_1123955_3.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/36cd1b55-c9aa-4023-9b06-15080453aed1_isr_1123955_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:18 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4f1b132d6ed860c1441de6b7e75b7f with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash b712fb2e2e2bbb3bb906a622cf7bca00 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/de71df5a-1e7f-4a66-a842-7e93aed7e0b4_characterizeImage_1123953_59.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/de71df5a-1e7f-4a66-a842-7e93aed7e0b4_characterizeImage_1123953_59.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0e3f851c-cc9c-4599-8444-3914a17e031b_isr_1123953_59.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/0e3f851c-cc9c-4599-8444-3914a17e031b_isr_1123953_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:18 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:19 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 322 try 0 failed parsl.dataflow.dflow INFO: Task 322 marked for retry parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash a1b0f9a2d8c46eb08134af70849f44b1 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 1 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:19 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:19 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:20 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:20 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:20 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:21 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 0 failed parsl.dataflow.dflow INFO: Task 1 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 2424cd0973a1a05fd83524ac06b799cc parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 1 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 0 failed parsl.dataflow.dflow INFO: Task 52 marked for retry parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 2e17cc24ee39e5b65e2951f1cd5142dd parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 1 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 0 failed parsl.dataflow.dflow INFO: Task 22 marked for retry parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash ca5e1a1ee94b8e173b279fbcd3986adf parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 1 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:21 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 79 try 0 failed parsl.dataflow.dflow INFO: Task 79 marked for retry parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 66a5c7d65ac0913a274af22ce100e4c6 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 1 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 31 try 0 failed parsl.dataflow.dflow INFO: Task 31 marked for retry parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash fd5e60ca0499ebf65b2043b1bb9c1455 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 1 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 109 try 0 failed parsl.dataflow.dflow INFO: Task 109 marked for retry parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash b34a0bb75ddd9d6447b8887a9393d9ba parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 1 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:22 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 169 try 0 failed parsl.dataflow.dflow INFO: Task 169 marked for retry parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 570769d5c76ec82bcac0109a091b40ca parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 1 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 277 try 0 failed parsl.dataflow.dflow INFO: Task 277 marked for retry parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash da4ced1143be1785788d565253a703af parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 1 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:22 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:22 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:23 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:23 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:23 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:24 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:24 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:24 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 184 try 0 failed parsl.dataflow.dflow INFO: Task 184 marked for retry parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash af8f3b018eb2730835430d25b15b9630 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 1 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36e1385d9f83d3b0c3df9e32297ab088 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash a8c23f316931b637a7912e16787a011f parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2a7063e7-3b59-4415-8b0b-e3ed5d09d692_isr_1124086_24.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/2a7063e7-3b59-4415-8b0b-e3ed5d09d692_isr_1124086_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:25 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:25 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:25 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:26 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:26 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:27 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:27 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:27 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:28 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:28 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 295 try 0 failed parsl.dataflow.dflow INFO: Task 295 marked for retry parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 977948b6d97373167b64a58a7610d16c parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 1 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 97 try 0 failed parsl.dataflow.dflow INFO: Task 97 marked for retry parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash f09886a46d581e9370265c46f1e5b8c1 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 1 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 253 try 0 failed parsl.dataflow.dflow INFO: Task 253 marked for retry parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 24c76bf06b06d647d038149601c67b11 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 1 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 307 try 0 failed parsl.dataflow.dflow INFO: Task 307 marked for retry parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 97ece9fc32bb40738a50be9516ad7430 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639add00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 1 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stderr 2024-03-07 00:40:38 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:40:38 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:41 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:41 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 235 try 0 failed parsl.dataflow.dflow INFO: Task 235 marked for retry parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash dd9d58b0a42d6eb58f8f214eafbfeac7 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639891c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 1 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 0 failed parsl.dataflow.dflow INFO: Task 118 marked for retry parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 2d022c45144f0d07758316012d546f8a parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 1 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:42 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:42 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:42 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:43 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:43 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:43 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:44 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:44 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:44 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:45 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:45 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:45 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:46 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:46 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:46 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:47 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:47 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:47 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:48 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:48 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:49 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:49 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:49 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:50 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:50 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:50 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:51 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:51 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:51 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:51 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:52 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:52 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:52 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:53 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:53 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:54 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:54 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:54 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:55 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:55 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:55 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:56 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:56 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:56 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:57 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:57 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:58 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:58 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:58 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:59 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fb0081648040be66b910d0f3b0ffe70 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash ff25dc2fac7ab8994269ae662569b9ee parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b7ead10f-8e52-4d9a-b3af-2aeb76c9da11_characterizeImage_1123953_47.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b7ead10f-8e52-4d9a-b3af-2aeb76c9da11_characterizeImage_1123953_47.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d321e201-d648-4e7e-a0da-46a2698c2fdc_isr_1123953_47.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/d321e201-d648-4e7e-a0da-46a2698c2fdc_isr_1123953_47.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32d85266cfc7e7ca04a3a11311c39549 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash f1101529b540d820d0d1eb1aafc134a1 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0ef6ffbc-2306-446f-9cce-e4dabfdf3805_characterizeImage_1123957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0ef6ffbc-2306-446f-9cce-e4dabfdf3805_characterizeImage_1123957_38.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/11c09840-db04-4dd7-9294-d6103cfaa535_isr_1123957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123957/11c09840-db04-4dd7-9294-d6103cfaa535_isr_1123957_38.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9c580887006680b9a37702bd261340 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash c26aa2901edd56434605a1b53a2f724f parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/12790f74-a77b-413f-a9d3-d723bb417209_isr_1124086_15.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/12790f74-a77b-413f-a9d3-d723bb417209_isr_1124086_15.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80374959410c3302f421a99514b2d87b with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 0684c7866c0b42b9b818b9bb98824cf1 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c8b673b5-7c10-4bb7-9bdd-67a6a04e85b0_characterizeImage_1123953_22.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c8b673b5-7c10-4bb7-9bdd-67a6a04e85b0_characterizeImage_1123953_22.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6372152b-e948-47cd-8d19-3a0e565f755c_isr_1123953_22.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/6372152b-e948-47cd-8d19-3a0e565f755c_isr_1123953_22.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7bc8e6e911ad450a21c06bc9d0b3c5c with result from task 354 parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:59 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash e3335aff30a081ce9082223a2aa02c3f parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ad5232f4-12ad-4361-8f93-1cced134879d_characterizeImage_1123953_56.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ad5232f4-12ad-4361-8f93-1cced134879d_characterizeImage_1123953_56.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/7b72d5d2-1d77-48cb-a77d-6f38ecc80577_isr_1123953_56.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123953/7b72d5d2-1d77-48cb-a77d-6f38ecc80577_isr_1123953_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:40:59 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:00 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bae29a0e241327a348510ed6c349e02 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash d58dd3a599c1456c2ade4ea45c2f0b23 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1caf4bd8-ca06-4c3b-bcc6-76f7134b98ca_characterizeImage_1123955_54.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1caf4bd8-ca06-4c3b-bcc6-76f7134b98ca_characterizeImage_1123955_54.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4326f2d6-a7ba-4f7e-8b43-1aa8c4bfd7d7_isr_1123955_54.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1123955/4326f2d6-a7ba-4f7e-8b43-1aa8c4bfd7d7_isr_1123955_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 PD 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:00 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93833ab19947a1e7aa33127677fffc83 with result from task 345 parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 PD 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:00 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 1d8308824eaf28aad553a8cef558f49b parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f623d912-ff39-450e-ab8b-a178bc400b1e_isr_1124086_50.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/f623d912-ff39-450e-ab8b-a178bc400b1e_isr_1124086_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 PD 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:01 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 PD 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:01 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b542812d5a891441db49d21f2efc0f2b with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash a2539746568cd4d0c7e943d5339c97eb parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/118fbaa3-3b3d-4162-a743-1d2b62a368e5_isr_1124086_43.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/isr/1124086/118fbaa3-3b3d-4162-a743-1d2b62a368e5_isr_1124086_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 PD 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:01 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 PD 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07b690fed1429b41a246d116eddbd89e with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash e5b4cc812a384c867a63dca4060509f2 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63906520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6553cab2-02e3-4468-8042-1e3d03517bdf_calibrate_1123953_3.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6553cab2-02e3-4468-8042-1e3d03517bdf_calibrate_1123953_3.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c19ff0ec-81ab-4653-a8cc-365952216c1c_characterizeImage_1123953_3.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c19ff0ec-81ab-4653-a8cc-365952216c1c_characterizeImage_1123953_3.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 2024-03-07 00:41:12 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:41:12 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 376 try 0 failed parsl.dataflow.dflow INFO: Task 376 marked for retry parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash acf2b064af358e67396ef9f14da38cac parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 1 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:14 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:14 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:15 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:16 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:16 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50f14de27581409ac7c61f02962b9d8a with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 085f090bd960466ec3a93b3c61b81f89 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/96b6e4b9-41be-494c-875d-e125b82c1475_calibrate_1123953_40.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/96b6e4b9-41be-494c-875d-e125b82c1475_calibrate_1123953_40.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a26e811b-2fa5-4dbf-b2bb-fe59c2f5e656_characterizeImage_1123953_40.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a26e811b-2fa5-4dbf-b2bb-fe59c2f5e656_characterizeImage_1123953_40.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3817f26de0617b8135932257eadf4654 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 3698b276ad4a6c04338b49bbfdf7d6ce parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/45234151-805b-48df-a5c7-98b4fc262198_calibrate_1123957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/45234151-805b-48df-a5c7-98b4fc262198_calibrate_1123957_62.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/07a21969-25c7-4d1e-9853-b3a9f79bf309_characterizeImage_1123957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/07a21969-25c7-4d1e-9853-b3a9f79bf309_characterizeImage_1123957_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:16 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:17 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:17 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:17 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cb322a28126ff984bce96bfb1d948df with result from task 319 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:17 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 60e569991278c2ee259b806a7cd023b4 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a2e7f4f4-c1f9-4c2e-95ea-0d5c1a0ec1ac_calibrate_1123955_14.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a2e7f4f4-c1f9-4c2e-95ea-0d5c1a0ec1ac_calibrate_1123955_14.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/af0e8036-1469-4ae5-be31-216897612814_characterizeImage_1123955_14.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/af0e8036-1469-4ae5-be31-216897612814_characterizeImage_1123955_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:18 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:18 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:18 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:19 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:19 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b6a8b42a61cc27894e92b39e6ead94f with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 4bf952a1551f924d6dc7b2295e695e94 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639604a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4eb87902-4477-4e10-bcb6-f10ea8829337_calibrate_1123957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4eb87902-4477-4e10-bcb6-f10ea8829337_calibrate_1123957_52.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/dd22ef8c-9a78-492c-81ae-e4b474a8daef_characterizeImage_1123957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/dd22ef8c-9a78-492c-81ae-e4b474a8daef_characterizeImage_1123957_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:19 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:19 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:20 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:20 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:20 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:20 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:20 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 214 try 0 failed parsl.dataflow.dflow INFO: Task 214 marked for retry parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash a704ab4d1dd1738c9d5fd0e11ca02d64 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 1 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:21 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 514cc06830849b05bc3ad2c9a3f4fc1f with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash adbe12072b5d76404ccdae40d13c8a94 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/445a1497-fad6-4ef1-a62c-86199764214e_calibrate_1123955_10.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/445a1497-fad6-4ef1-a62c-86199764214e_calibrate_1123955_10.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3798d24b-1094-4cdf-8eb2-52d8c67e36cc_characterizeImage_1123955_10.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3798d24b-1094-4cdf-8eb2-52d8c67e36cc_characterizeImage_1123955_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:21 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c21c21a765de970dcf784e8991372094 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash befb906dc3e3d9272113f719d8bae610 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/42f4adb4-20d1-4d75-8c3e-dc0651603f53_characterizeImage_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/42f4adb4-20d1-4d75-8c3e-dc0651603f53_characterizeImage_1123953_31.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f51fa2699233e9b7ddc9ab4fb3ae0a3e with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 94113a165b0dfccfe817a46461c0e16b parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2e315c1f-b53e-4814-ac93-b0f427aeb4c6_calibrate_1123955_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2e315c1f-b53e-4814-ac93-b0f427aeb4c6_calibrate_1123955_41.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35da64ce-8027-439f-a5e5-85a89e8db873_characterizeImage_1123955_41.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35da64ce-8027-439f-a5e5-85a89e8db873_characterizeImage_1123955_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:21 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 421 try 0 failed parsl.dataflow.dflow INFO: Task 421 marked for retry parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 9d330e4c97e2b8d9486e21f7e23e1c68 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 1 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:22 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:22 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 638cfeddb262ffdabf048ece5d9fdfdb with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash f89bd548be9c52c9af04d8069b5a7d1e parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7021b942-0e24-4fe4-83e8-e0b028b5cdac_calibrate_1123953_62.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7021b942-0e24-4fe4-83e8-e0b028b5cdac_calibrate_1123953_62.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e4a80988-19dd-44b4-b40b-04ea2abacfe4_characterizeImage_1123953_62.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e4a80988-19dd-44b4-b40b-04ea2abacfe4_characterizeImage_1123953_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:22 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:22 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:23 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 280 try 0 failed parsl.dataflow.dflow INFO: Task 280 marked for retry parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 7b6ff2c8ebaa5d2bb20c7fa2e2fb1a4b parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 1 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:23 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:23 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:23 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29f00f12b2f4a62826b89a1ca40aa4ab with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 8e24c4528b28ada02f85e1a8adb7de09 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b509c223-503d-45db-8ecb-b4ab098c66a4_calibrate_1123955_6.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b509c223-503d-45db-8ecb-b4ab098c66a4_calibrate_1123955_6.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c64c503d-d91f-410c-9575-6ade0fd92dda_characterizeImage_1123955_6.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c64c503d-d91f-410c-9575-6ade0fd92dda_characterizeImage_1123955_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:23 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:23 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7bae79fa12181cd0eb69565daf465d6 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 780246211932e09fe586d21dcdc4f47a parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/09837c47-f4ba-4016-a570-3a7d77d5d52e_calibrate_1123957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/09837c47-f4ba-4016-a570-3a7d77d5d52e_calibrate_1123957_29.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/25311660-4478-4873-b34d-facacf7b54c3_characterizeImage_1123957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/25311660-4478-4873-b34d-facacf7b54c3_characterizeImage_1123957_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:23 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:24 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:24 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:24 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8f76c6f5fad4c8b246f9809be7f2be2 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 35db422d28192f7155de1425a9d73b32 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6f158d77-2485-4b50-adee-e0cc4c4f3450_calibrate_1123953_1.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6f158d77-2485-4b50-adee-e0cc4c4f3450_calibrate_1123953_1.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1311e7a1-3496-4525-8b63-de772d437b74_characterizeImage_1123953_1.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1311e7a1-3496-4525-8b63-de772d437b74_characterizeImage_1123953_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:24 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:24 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:25 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e62f4e94db37e716d090744a577c3bb with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 68dd5d258fcf24e4b3782d931a500c43 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6e2a25c3-37c8-4d5b-b01e-1353cac44fb6_calibrate_1123953_46.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6e2a25c3-37c8-4d5b-b01e-1353cac44fb6_calibrate_1123953_46.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb2b916a-50ca-42ed-be8a-8b77d0a36650_characterizeImage_1123953_46.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb2b916a-50ca-42ed-be8a-8b77d0a36650_characterizeImage_1123953_46.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd57927a6d29538e25cd3858b9d67bfe with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 22f6f6734a1f854f7ad5b365b97a9420 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3faae82b-d0b8-49c0-9af6-73c7b71aa0f1_calibrate_1123955_23.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3faae82b-d0b8-49c0-9af6-73c7b71aa0f1_calibrate_1123955_23.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d2242eed-bbd2-4536-b223-23126326bb3a_characterizeImage_1123955_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d2242eed-bbd2-4536-b223-23126326bb3a_characterizeImage_1123955_23.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 232 try 0 failed parsl.dataflow.dflow INFO: Task 232 marked for retry parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 4468ec30f1497c7fc5358bf13609d8ed parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 1 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:25 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:25 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:25 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:25 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:25 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:26 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc62610edffd82c53c3d92f459aa71c with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 347034fa272c219a2867ef754dd8ce9d parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6c82a894-d6b7-4409-b809-f317aba237b0_calibrate_1123955_59.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6c82a894-d6b7-4409-b809-f317aba237b0_calibrate_1123955_59.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ca3ee79e-e3ff-400c-9be4-3090244d14f1_characterizeImage_1123955_59.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ca3ee79e-e3ff-400c-9be4-3090244d14f1_characterizeImage_1123955_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:26 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aa738d36669fa89a8c401581fa7c180 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 612137701453aba31e72ca55512a9e8a parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f8f11e-ec7c-4b2e-9b10-89302a0eb9e5_calibrate_1123955_32.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f8f11e-ec7c-4b2e-9b10-89302a0eb9e5_calibrate_1123955_32.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c63ce9e0-9311-4751-8553-4c4e79e69062_characterizeImage_1123955_32.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c63ce9e0-9311-4751-8553-4c4e79e69062_characterizeImage_1123955_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:26 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:27 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:27 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:27 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:28 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95dc80d011ad5fa5b302e04c0483d46 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 8a4edb35c52e6bdd6ae2e0717e15e769 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/49c2cc46-d244-43bd-b734-02e09549a7c8_calibrate_1123957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/49c2cc46-d244-43bd-b734-02e09549a7c8_calibrate_1123957_33.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/87bc1aa0-e419-43af-9f3d-3cc732ee252b_characterizeImage_1123957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/87bc1aa0-e419-43af-9f3d-3cc732ee252b_characterizeImage_1123957_33.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccac0af4999148de8121aeccf4aafa16 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash d4f9a535b31ddf576060ba4d8dad174e parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/40c11632-5f6f-4f23-9b74-6b4f0f64dee0_calibrate_1123955_5.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/40c11632-5f6f-4f23-9b74-6b4f0f64dee0_calibrate_1123955_5.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/13115233-2acc-4c86-85fe-15976760f33e_characterizeImage_1123955_5.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/13115233-2acc-4c86-85fe-15976760f33e_characterizeImage_1123955_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:28 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2068c067b9ee7d437379b30f03dff674 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash caf9b8746a6e5379d78b2aaac687db2c parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639acb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/63949d4a-0945-4b8f-9f12-4abacbf57c7c_calibrate_1123955_36.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/63949d4a-0945-4b8f-9f12-4abacbf57c7c_calibrate_1123955_36.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/4c5c1725-aba8-4257-b381-91c77aaa542d_characterizeImage_1123955_36.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/4c5c1725-aba8-4257-b381-91c77aaa542d_characterizeImage_1123955_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:28 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:29 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 385 try 0 failed parsl.dataflow.dflow INFO: Task 385 marked for retry parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 23909fe7c63d1a66812555eb529fbcc4 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 1 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49029c67ffec79a18d4f5cd92e2cdcf9 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 2061d9d9fcb7184328b7ef0b4cc5e0de parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/f93a04d4-5cbb-482a-9f04-b085c55d3f02_calibrate_1123957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/f93a04d4-5cbb-482a-9f04-b085c55d3f02_calibrate_1123957_19.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/cc490b81-e030-4f98-9381-4e564f8449ea_characterizeImage_1123957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/cc490b81-e030-4f98-9381-4e564f8449ea_characterizeImage_1123957_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:29 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:29 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f59b805aae2342559d4fcd5e2e07da with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 7e60938d64a53629102c397040f026cb parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a6c9c2ae-bfce-40c0-9793-fa47a388637d_characterizeImage_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a6c9c2ae-bfce-40c0-9793-fa47a388637d_characterizeImage_1123957_31.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c79745b3527d1ca62b56280c338a837a with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 2a4dc0a5ba90cd61a90feedc658a9dd5 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d7909e57-4f6f-4072-8362-874e3c4138e1_calibrate_1123953_9.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d7909e57-4f6f-4072-8362-874e3c4138e1_calibrate_1123953_9.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8e3cdf69-c54c-4b4d-b041-6d422dcc2e40_characterizeImage_1123953_9.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8e3cdf69-c54c-4b4d-b041-6d422dcc2e40_characterizeImage_1123953_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:30 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b1aed6ee546021ff4961dd3ea47740f with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 4c5e3bbface9d92724222ad0d3eaf0a7 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9cf098bf-025e-4942-8128-e479cc1dd36f_calibrate_1123957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9cf098bf-025e-4942-8128-e479cc1dd36f_calibrate_1123957_16.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f505d930-556f-4df1-a196-1398c820b952_characterizeImage_1123957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f505d930-556f-4df1-a196-1398c820b952_characterizeImage_1123957_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47a3981b36b1a24914596af76d99ec39 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash c09dca698effbf89db7ba0b4d78d7035 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ff423ae4-fae2-4940-b577-cf353ea6de18_characterizeImage_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ff423ae4-fae2-4940-b577-cf353ea6de18_characterizeImage_1123955_19.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d93694a3014b23863064ab4d6357060 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash cb52037a0a4744cd8457213b0a84d2e8 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e0a514f6-9ba7-43a5-b958-134c15960edd_characterizeImage_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e0a514f6-9ba7-43a5-b958-134c15960edd_characterizeImage_1123955_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 310 try 0 failed parsl.dataflow.dflow INFO: Task 310 marked for retry parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash a34ef020eff8f6f050f0786277892043 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 1 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 409 try 0 failed parsl.dataflow.dflow INFO: Task 409 marked for retry parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 1c731bfbed0ebde45e6ab1f1e4995d36 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628087c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 1 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 352eb8e94612665e98f6a5cec527867f with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash a3d5beebae92e410569e705713e10c6b parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/864bd408-4625-4bfc-9a69-7b149463e7ec_calibrate_1123953_6.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/864bd408-4625-4bfc-9a69-7b149463e7ec_calibrate_1123953_6.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3f2c6947-429c-4c91-9eef-d44d2d69c40b_characterizeImage_1123953_6.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3f2c6947-429c-4c91-9eef-d44d2d69c40b_characterizeImage_1123953_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 547 try 0 failed parsl.dataflow.dflow INFO: Task 547 marked for retry parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash c28f4bf5b893737d3b073661120fb2ec parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 1 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b17bad51a64c33f6365c4ab48604e4d5 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 99b36eb79dee1a263585f92e38446d7c parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a25caccb-d81c-448a-9cb0-8714cd7d284f_calibrate_1123953_28.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a25caccb-d81c-448a-9cb0-8714cd7d284f_calibrate_1123953_28.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2acb59ba-73d1-43d8-a7e6-7fced8d91e62_characterizeImage_1123953_28.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2acb59ba-73d1-43d8-a7e6-7fced8d91e62_characterizeImage_1123953_28.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3aecd33397369753ed733fc71c10ec2 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash ca5263249ecb940a11f8e6b0faf7bc64 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0f5c6571-f910-465a-9399-5a0dc22d907d_calibrate_1123955_47.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0f5c6571-f910-465a-9399-5a0dc22d907d_calibrate_1123955_47.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bd8e13-b190-4f25-9d3b-c6762590da44_characterizeImage_1123955_47.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bd8e13-b190-4f25-9d3b-c6762590da44_characterizeImage_1123955_47.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5202936de168c8617faffe64655fa2c with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 8ddaa8a567267b4f9ef341c04243f6b0 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/25c8ebc9-02ca-4af6-8b06-83c94cf504ff_calibrate_1123953_42.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/25c8ebc9-02ca-4af6-8b06-83c94cf504ff_calibrate_1123953_42.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7501502e-23c4-4452-9bcf-c3e5ecb3ed63_characterizeImage_1123953_42.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7501502e-23c4-4452-9bcf-c3e5ecb3ed63_characterizeImage_1123953_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 568 try 0 failed parsl.dataflow.dflow INFO: Task 568 marked for retry parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash a62a0ec46601102648cc5a82724b1af0 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 1 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b30c485162c382cf68a4ee2e07242f98 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 75b210d20d72db90488b8495c9844184 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a773cb8-d36d-4691-bc6c-8b8087939595_calibrate_1123953_30.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a773cb8-d36d-4691-bc6c-8b8087939595_calibrate_1123953_30.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/78f640b5-9e5b-43c2-8ae5-89b8c71c4155_characterizeImage_1123953_30.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/78f640b5-9e5b-43c2-8ae5-89b8c71c4155_characterizeImage_1123953_30.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3474378a0deb6d23ff4a7e1a049e3e1e with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 07efc31d3f502bf60e4e6ce087c45efe parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bff37879-148b-464b-97aa-6d8ac8a890d6_calibrate_1123955_27.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bff37879-148b-464b-97aa-6d8ac8a890d6_calibrate_1123955_27.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c2e8c105-d4f2-45f5-bddb-36e6b3923790_characterizeImage_1123955_27.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c2e8c105-d4f2-45f5-bddb-36e6b3923790_characterizeImage_1123955_27.stderr parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 247 parsl.dataflow.strategy DEBUG: Executor multi has 247 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cec8d2ee48a4b663cd4d3406679572b with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash fed91f899d13206c1e6e3461e52cd20c parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ea0f96f5-11eb-4921-b63a-4856e5843dca_calibrate_1123955_40.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ea0f96f5-11eb-4921-b63a-4856e5843dca_calibrate_1123955_40.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e055fa9-7c2c-4c6b-b10c-993c57867ec2_characterizeImage_1123955_40.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e055fa9-7c2c-4c6b-b10c-993c57867ec2_characterizeImage_1123955_40.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52e3fd16bd6f4119e0da51d2452d97e2 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 0f655eaa02eee626544996ea7dd4119a parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/deeec811-24dd-47d5-9e90-cc47ed67c0bc_calibrate_1123953_44.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/deeec811-24dd-47d5-9e90-cc47ed67c0bc_calibrate_1123953_44.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2a543a3b-df48-4998-93bb-8a7b2ad48f47_characterizeImage_1123953_44.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2a543a3b-df48-4998-93bb-8a7b2ad48f47_characterizeImage_1123953_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 526 try 0 failed parsl.dataflow.dflow INFO: Task 526 marked for retry parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash e6c8f7a701148714415480c763f89556 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 1 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fb53547b2a99fc9dde58c38ff5fd974 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash ff9ec2860bb518ba5119b8f15a002d63 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639889a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c476ec8c-cda0-4193-b4c6-52d686f75fc0_calibrate_1123957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c476ec8c-cda0-4193-b4c6-52d686f75fc0_calibrate_1123957_41.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/da997a21-7397-495a-afb0-168b2dbc61e1_characterizeImage_1123957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/da997a21-7397-495a-afb0-168b2dbc61e1_characterizeImage_1123957_41.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4087e0d31e0768cf60a9c40165f52e84 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash b446d44ed1dd4dab0366f4aca8a4d745 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63907420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bf460135-27cb-4f2b-b672-149830f37e48_calibrate_1123953_13.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bf460135-27cb-4f2b-b672-149830f37e48_calibrate_1123953_13.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/415290e1-e167-4e15-99ae-7729d18e45cd_characterizeImage_1123953_13.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/415290e1-e167-4e15-99ae-7729d18e45cd_characterizeImage_1123953_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 391 try 0 failed parsl.dataflow.dflow INFO: Task 391 marked for retry parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 31b84a3b71164a0e0a55d387eb617bd3 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 1 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 463 try 0 failed parsl.dataflow.dflow INFO: Task 463 marked for retry parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash dc6f9dbb7f1a31cbfe08969ac84f85d4 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 1 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91e9f79287f99ecde48c83176c358088 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 0634b741da09cdc6717f20bf940a46ce parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5dec0fd6-19a5-40f2-a21c-97a554c53f27_calibrate_1123955_13.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5dec0fd6-19a5-40f2-a21c-97a554c53f27_calibrate_1123955_13.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1facdfad-2470-4a61-95da-031c7515fe6b_characterizeImage_1123955_13.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1facdfad-2470-4a61-95da-031c7515fe6b_characterizeImage_1123955_13.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb35ffbcd89790a85fc7a79a143f5433 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash a8f5df28e2489c118a1a8cec1370c77b parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/331daff7-f003-4e52-a7e8-a6eaadc97ee7_calibrate_1123957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/331daff7-f003-4e52-a7e8-a6eaadc97ee7_calibrate_1123957_15.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/9082b2c2-4cde-405a-9d2f-8a654059ac08_characterizeImage_1123957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/9082b2c2-4cde-405a-9d2f-8a654059ac08_characterizeImage_1123957_15.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 541 try 0 failed parsl.dataflow.dflow INFO: Task 541 marked for retry parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash eac9364a230113d22ceff5dd36acd291 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628991c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 1 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 448 try 0 failed parsl.dataflow.dflow INFO: Task 448 marked for retry parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 2c1c88107ce808a9d707329c2b4d7221 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 1 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26490521d4c66baa743e9d5109da36c6 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 002a2115097818868b99c5986819b9cf parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c09fabe0-1f9e-4b33-8427-77d1cb3e69a3_calibrate_1123957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c09fabe0-1f9e-4b33-8427-77d1cb3e69a3_calibrate_1123957_42.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90cc17ed-6e38-4a11-b10c-2a20b4a8c371_characterizeImage_1123957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90cc17ed-6e38-4a11-b10c-2a20b4a8c371_characterizeImage_1123957_42.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 927e529a27af16df85be97c9735f3127 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 8cf41b7d0e7d7081aed66514e3ecd854 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63960180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cb1705d5-ddf0-4514-99f0-0d172719f1da_calibrate_1123957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cb1705d5-ddf0-4514-99f0-0d172719f1da_calibrate_1123957_10.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1f33aa00-e033-4b77-a719-79a04dac9b4e_characterizeImage_1123957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1f33aa00-e033-4b77-a719-79a04dac9b4e_characterizeImage_1123957_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 553 try 0 failed parsl.dataflow.dflow INFO: Task 553 marked for retry parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash d60753152da78404cb8d5f35dbe2f2a1 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 1 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 394 try 0 failed parsl.dataflow.dflow INFO: Task 394 marked for retry parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 8f7ec507cde9853dd144343303d12d8c parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 1 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b86abb6c62c2d790e33f29b8aa34b757 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash b299f921ad60de691d19de9c6e6cc80b parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2c01fd70-b692-4f87-92e4-4780e22f124a_calibrate_1123953_8.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2c01fd70-b692-4f87-92e4-4780e22f124a_calibrate_1123953_8.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c3c1caf4-ce99-4639-8ca2-888c1baafb28_characterizeImage_1123953_8.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c3c1caf4-ce99-4639-8ca2-888c1baafb28_characterizeImage_1123953_8.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 665f7d1c23c3e652128c94b6f6984062 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash b0d5fadb1d1d45f48e7743882fc07ac6 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/df04b87c-d5b0-4a4b-a023-0c59188ac4f3_calibrate_1123953_36.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/df04b87c-d5b0-4a4b-a023-0c59188ac4f3_calibrate_1123953_36.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d7cd7a10-8e9d-4e02-9cdf-740b4a4f6c69_characterizeImage_1123953_36.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d7cd7a10-8e9d-4e02-9cdf-740b4a4f6c69_characterizeImage_1123953_36.stderr 2024-03-07 00:41:45 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:41:45 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:45 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0182ff3eeca0eb6ea62e74bade1ed6ab with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 838118bd48b06d2597b0237c03f83db5 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d58872ce-9d5e-4beb-ab66-a91433fa9d12_calibrate_1123955_52.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d58872ce-9d5e-4beb-ab66-a91433fa9d12_calibrate_1123955_52.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/06f4515a-6953-49b5-8680-5422e5c0dc8b_characterizeImage_1123955_52.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/06f4515a-6953-49b5-8680-5422e5c0dc8b_characterizeImage_1123955_52.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 538 try 0 failed parsl.dataflow.dflow INFO: Task 538 marked for retry parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash d6df22c1985fdff01a2d8c71f3363c5a parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 1 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 442 try 0 failed parsl.dataflow.dflow INFO: Task 442 marked for retry parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 1f93203f4e9f414f54589472d4f8fad8 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 1 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:47 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f991d54fa6ac8ba346cb44d1b4b16758 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash f818e813d99a66fd0409b16681867934 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0459b698-3887-4923-a003-ef993625a7d9_calibrate_1123955_11.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0459b698-3887-4923-a003-ef993625a7d9_calibrate_1123955_11.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/084f2898-9d24-4e5a-a5c8-c8a3c4e0b948_characterizeImage_1123955_11.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/084f2898-9d24-4e5a-a5c8-c8a3c4e0b948_characterizeImage_1123955_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:48 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 523 try 0 failed parsl.dataflow.dflow INFO: Task 523 marked for retry parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash a9d1832a360c9a2c6233bb2591c7c2cb parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 1 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9132c49ae6b966707a495e215314c8e0 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash d94d28cb4f3a38f28530210b64b9030c parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/34eea4f7-5188-478d-8026-baebebbfb025_calibrate_1123957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/34eea4f7-5188-478d-8026-baebebbfb025_calibrate_1123957_3.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe57bfb-3e40-4587-9478-56946e34f661_characterizeImage_1123957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe57bfb-3e40-4587-9478-56946e34f661_characterizeImage_1123957_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 193 try 0 failed parsl.dataflow.dflow INFO: Task 193 marked for retry parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash c3d49b98236bbd759e12c588adbade8e parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639625c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 1 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18862d370194de3a4c0a37ed109c4356 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 02cdd02d74c3bd19df59a9bf7fccea03 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/88776509-1190-4108-8e43-0d80bff54982_calibrate_1123955_48.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/88776509-1190-4108-8e43-0d80bff54982_calibrate_1123955_48.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64e9ceaa-ecb2-4edb-bf41-79615b53d5ad_characterizeImage_1123955_48.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64e9ceaa-ecb2-4edb-bf41-79615b53d5ad_characterizeImage_1123955_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:49 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:50 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 343 try 0 failed parsl.dataflow.dflow INFO: Task 343 marked for retry parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash ee6a5e8a80bc064569e6f60791c72a01 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 1 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:50 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:50 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:51 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:51 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a6906477b5339ad7873a35dd6fc0ded with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 5dc4f1fb7fdb458c0289fd9d4a316ac7 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/2c82422a-7a89-4f4f-bcf6-40931b789c58_calibrate_1123957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/2c82422a-7a89-4f4f-bcf6-40931b789c58_calibrate_1123957_54.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0bbfb682-0e2d-4951-b8a6-b896fad31e6e_characterizeImage_1123957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0bbfb682-0e2d-4951-b8a6-b896fad31e6e_characterizeImage_1123957_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 496 try 0 failed parsl.dataflow.dflow INFO: Task 496 marked for retry parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 04698d798d4bd343184502558f343f84 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 1 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9096c5fb8acc4699c470f9e8ef75b67 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 2939f0d1ba6dd7bbf5925a7f55bf6c2a parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/25f6b895-a656-48cb-b299-587c127707c2_calibrate_1123955_12.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/25f6b895-a656-48cb-b299-587c127707c2_calibrate_1123955_12.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2f8812e2-8596-4acf-b252-a8db3480e925_characterizeImage_1123955_12.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2f8812e2-8596-4acf-b252-a8db3480e925_characterizeImage_1123955_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:51 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab8e68b25cac6435efb0da8a1f3c16d1 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash ae5bfa09a3fdfda730dba7034c2f12de parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ad4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cfcc7c8d-ede0-4c4a-85f7-57e23d60901d_calibrate_1123957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cfcc7c8d-ede0-4c4a-85f7-57e23d60901d_calibrate_1123957_43.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/340106fb-a8cd-403c-a58e-bde03cb5fef6_characterizeImage_1123957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/340106fb-a8cd-403c-a58e-bde03cb5fef6_characterizeImage_1123957_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:52 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:52 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba6c4b4aeab6b804751c7961a157140 with result from task 328 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:53 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 4dc0b6d9ca23e0fcc2fe7ced4321cb4b parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a85fe8a3-9d3f-468e-b62b-02d6d620798c_calibrate_1123957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a85fe8a3-9d3f-468e-b62b-02d6d620798c_calibrate_1123957_20.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c9efe260-ff2f-435e-9835-bd69c572aec3_characterizeImage_1123957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c9efe260-ff2f-435e-9835-bd69c572aec3_characterizeImage_1123957_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:53 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:53 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73e70045573d8229e55d595337359440 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 20d23be6a486b74a111e51c8d54409a9 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a172caa0-21e4-4704-8748-f2579dd0c599_calibrate_1123955_17.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a172caa0-21e4-4704-8748-f2579dd0c599_calibrate_1123955_17.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8071d203-70e1-4f68-ac0e-6fc117a75477_characterizeImage_1123955_17.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8071d203-70e1-4f68-ac0e-6fc117a75477_characterizeImage_1123955_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:54 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:54 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3afc849b2c8e09e2de489bd5038b7daa with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 12b04df649440141125304743af12c68 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3abc537f-205b-41b9-bb93-069ec7ee0459_calibrate_1123953_37.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3abc537f-205b-41b9-bb93-069ec7ee0459_calibrate_1123953_37.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/94938c59-63b7-4fee-8b6d-d30902073539_characterizeImage_1123953_37.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/94938c59-63b7-4fee-8b6d-d30902073539_characterizeImage_1123953_37.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29da1a067e194106f9ab7ba7fbeeb7ed with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash c73243fc7de9187a81325a8161922be2 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ac220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d8dd1eae-f213-4765-96c5-e9c8f83cba43_calibrate_1123953_4.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d8dd1eae-f213-4765-96c5-e9c8f83cba43_calibrate_1123953_4.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1b1483a1-7118-414e-b47f-c2a187935ea0_characterizeImage_1123953_4.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1b1483a1-7118-414e-b47f-c2a187935ea0_characterizeImage_1123953_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:54 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:55 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84bfccfa0c8b87bdea82c39522e90c55 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 3bc608979ada3045d63fe30df481041d parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/98fb946a-8981-4af8-b665-69199fe5abd6_calibrate_1123957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/98fb946a-8981-4af8-b665-69199fe5abd6_calibrate_1123957_40.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fad9b6f1-0456-47e0-abf7-2ce7d885c528_characterizeImage_1123957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fad9b6f1-0456-47e0-abf7-2ce7d885c528_characterizeImage_1123957_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:55 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:55 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:56 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:56 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:56 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:57 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 631 try 0 failed parsl.dataflow.dflow INFO: Task 631 marked for retry parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 2bf0cbd27c0283b36a0dad924269a652 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 1 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:57 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:57 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9150f40e8bc9d82f6494b4660a393b5 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 4e28559432127ee3e8dc8db9e3e3428d parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63961da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b18c4571-ae32-47ac-95e5-cc58a9e36f1a_calibrate_1123957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b18c4571-ae32-47ac-95e5-cc58a9e36f1a_calibrate_1123957_50.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c2e0c1c5-8b7e-44d5-86f3-5dfdd437fd22_characterizeImage_1123957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c2e0c1c5-8b7e-44d5-86f3-5dfdd437fd22_characterizeImage_1123957_50.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 718fb1e11434d7685895ba549470d5d6 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 0080da7e97d9c1d2a43ecd2e4ce97377 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b3e3ff9-a4be-4d89-a02f-7ebfc583a8ee_calibrate_1123953_53.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b3e3ff9-a4be-4d89-a02f-7ebfc583a8ee_calibrate_1123953_53.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/9c5d064b-9329-4287-ad73-a2dfca89743d_characterizeImage_1123953_53.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/9c5d064b-9329-4287-ad73-a2dfca89743d_characterizeImage_1123953_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:58 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:58 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 508 try 0 failed parsl.dataflow.dflow INFO: Task 508 marked for retry parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 454abd4bc0d448a57cdb310375233ce8 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 1 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:59 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:59 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3afbb0e63e8066cd1e9037bdfbf3b59e with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 5aeea6089ac98023aeb4c6309834e2c6 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ac540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/95084901-507b-49b7-b803-708a338b33c8_calibrate_1123953_33.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/95084901-507b-49b7-b803-708a338b33c8_calibrate_1123953_33.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/24ef8443-c405-49ee-8e6a-721d05fbeaaf_characterizeImage_1123953_33.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/24ef8443-c405-49ee-8e6a-721d05fbeaaf_characterizeImage_1123953_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:41:59 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:00 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c975a55f428d35f779199589d5d1d111 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash aa929eaf7c125c07f54ddbbc6eb7e1b9 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/42cbffa2-b430-410f-9283-309ae8e04b9f_calibrate_1123957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/42cbffa2-b430-410f-9283-309ae8e04b9f_calibrate_1123957_47.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e4823db7-8356-486d-9ab0-0f57788687fb_characterizeImage_1123957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e4823db7-8356-486d-9ab0-0f57788687fb_characterizeImage_1123957_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:00 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 610 try 0 failed parsl.dataflow.dflow INFO: Task 610 marked for retry parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash bd20a3c757288cac0aab6495a16f2a4b parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 1 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b44e6c05bc55bd8484d2be3ec1d83ba6 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 9bcc33a4f38ca75545a81815580759b9 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c9fb6a76-62b4-4092-9f46-ce288b155512_characterizeImage_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c9fb6a76-62b4-4092-9f46-ce288b155512_characterizeImage_1123953_61.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0248b0e86b2c3ebed16bccece728f102 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 7ef6bd765d4f2c2293ebab15b725a692 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/6914ae6d-228e-41f8-aa14-98ee6090f90d_calibrate_1123957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/6914ae6d-228e-41f8-aa14-98ee6090f90d_calibrate_1123957_13.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c1cc1fcb-0aed-436c-9cf0-b217b12a241f_characterizeImage_1123957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c1cc1fcb-0aed-436c-9cf0-b217b12a241f_characterizeImage_1123957_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:00 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:01 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:01 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 529 try 0 failed parsl.dataflow.dflow INFO: Task 529 marked for retry parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash f7af245d36596ef4c4f9435c61894e2c parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628982c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 1 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 604 try 0 failed parsl.dataflow.dflow INFO: Task 604 marked for retry parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash a6034ae8e3b23f47abdc90cc9d93c832 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 1 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:02 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 634 try 0 failed parsl.dataflow.dflow INFO: Task 634 marked for retry parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 86f043d6c9387ab64c67a62550153e19 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 1 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:02 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada16985ce8dc4386a7ad5f4f674ba2d with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash eb2faff98240ad1860b69f4417e0f385 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e01252c5-434b-43d7-88d4-6f883730cddd_calibrate_1123953_24.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e01252c5-434b-43d7-88d4-6f883730cddd_calibrate_1123953_24.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d05c3d7b-9649-4e94-aae8-bd804484786c_characterizeImage_1123953_24.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d05c3d7b-9649-4e94-aae8-bd804484786c_characterizeImage_1123953_24.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 475 try 0 failed parsl.dataflow.dflow INFO: Task 475 marked for retry parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 0f90651eb258afdb3d48d4b209f202d2 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 1 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:02 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:03 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:03 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:03 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:04 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:04 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cb4dfd39931dfe77a8999685fbf97a8 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 530a20dbf36bf2efd3976d9842f478e4 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63988360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/053082a8-b406-4769-91c0-8d4aa781a32a_calibrate_1123955_18.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/053082a8-b406-4769-91c0-8d4aa781a32a_calibrate_1123955_18.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5a3e6023-215a-49ad-92d8-ee8c7495ad1f_characterizeImage_1123955_18.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5a3e6023-215a-49ad-92d8-ee8c7495ad1f_characterizeImage_1123955_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:04 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 556 try 0 failed parsl.dataflow.dflow INFO: Task 556 marked for retry parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 4b0fb9474321991e8acbdb77e04dd5de parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 1 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75ac6dac25655bb7b97dcc33c9ae43c3 with result from task 190 parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:05 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 5c2abcc8cd70f95f7e2f3d5f19da8381 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/16c15251-3a71-4ff1-bc20-0b4c81e4e375_calibrate_1123957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/16c15251-3a71-4ff1-bc20-0b4c81e4e375_calibrate_1123957_22.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a73e210d-12a5-404f-a7dc-05b98dc9e9f1_characterizeImage_1123957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a73e210d-12a5-404f-a7dc-05b98dc9e9f1_characterizeImage_1123957_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:05 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22df70d5f0dfadde174606e652bcd90d with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 6b843147e6d565e7611a6cb5d4899f97 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/4367cb71-7280-465d-9cd9-927e0040c001_calibrate_1123955_53.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/4367cb71-7280-465d-9cd9-927e0040c001_calibrate_1123955_53.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/34831632-7e23-4379-8b6b-d3fb1a569511_characterizeImage_1123955_53.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/34831632-7e23-4379-8b6b-d3fb1a569511_characterizeImage_1123955_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:05 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:06 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 954d0240d2b2e2c109e45382c3a20fcf with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 496dd38d4b50b8b80b95f5d9b2dac966 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/40df4906-d547-4f56-b0ac-c7e0cf5f0309_calibrate_1123957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/40df4906-d547-4f56-b0ac-c7e0cf5f0309_calibrate_1123957_7.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/34cb97ce-2df8-4ccc-bd3c-a5d467b3d484_characterizeImage_1123957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/34cb97ce-2df8-4ccc-bd3c-a5d467b3d484_characterizeImage_1123957_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 325 try 0 failed parsl.dataflow.dflow INFO: Task 325 marked for retry parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 520d9b2064cccd9ca720a96a980ca4c8 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639aefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 1 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:06 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 440e0b0fbd9f1bb20ad74b53942c82b4 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash e730e01d8b1b7017a14f74a53a37f43f parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5e0b7c4e-38c4-4358-994e-c8253e7acca1_calibrate_1123955_60.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5e0b7c4e-38c4-4358-994e-c8253e7acca1_calibrate_1123955_60.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64fe7f59-3d65-441a-b012-9835b2961eb3_characterizeImage_1123955_60.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/64fe7f59-3d65-441a-b012-9835b2961eb3_characterizeImage_1123955_60.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de398aab062d48318bf974200a455e07 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 9f353dc895b460be666d92d69025d249 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639ae160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/38ea1ffd-8081-4408-8057-8bde56d9c1b5_calibrate_1123957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/38ea1ffd-8081-4408-8057-8bde56d9c1b5_calibrate_1123957_51.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6e0527d5-63bc-4083-aa6e-15e2e1dac337_characterizeImage_1123957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6e0527d5-63bc-4083-aa6e-15e2e1dac337_characterizeImage_1123957_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:06 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:07 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:07 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:07 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:08 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:08 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:08 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c20374f686dff73457fa0bf320e2a6e1 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash f7c89ca0226815d96c9a84960e0ea621 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b72b3486-bed6-4ae9-b952-db21dbb26e74_calibrate_1123955_39.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b72b3486-bed6-4ae9-b952-db21dbb26e74_calibrate_1123955_39.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/02d31865-d1d5-4a90-818a-438ccd63cfef_characterizeImage_1123955_39.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/02d31865-d1d5-4a90-818a-438ccd63cfef_characterizeImage_1123955_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:08 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 685 try 0 failed parsl.dataflow.dflow INFO: Task 685 marked for retry parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 1eb5af4c7b82d754d1de710d683f7fc7 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 1 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:09 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 330247b8434295c6a81a393b8408beaa with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 2d9c2e2365f5f126ac92b1bb10f522ad parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/37f2ce68-cf0d-4ced-b092-4ccb0040739c_calibrate_1123957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/37f2ce68-cf0d-4ced-b092-4ccb0040739c_calibrate_1123957_58.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c95ff939-d0c8-4e49-a36a-915253463efa_characterizeImage_1123957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/c95ff939-d0c8-4e49-a36a-915253463efa_characterizeImage_1123957_58.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53928c8a6a3b0b685590100addc781f3 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash bc307e51bb1a4c94cf2107f8acaa59d4 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/374739bb-fbc4-4cd3-a2a7-52f3fd5d7ce9_calibrate_1123955_43.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/374739bb-fbc4-4cd3-a2a7-52f3fd5d7ce9_calibrate_1123955_43.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e2f52acd-ac81-41df-9fb7-b3394e9da25c_characterizeImage_1123955_43.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e2f52acd-ac81-41df-9fb7-b3394e9da25c_characterizeImage_1123955_43.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ab0ff6add0cb29fe2ee623a7c3d94a with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 3ba3de698432dbb4d5ae5edbd4d17926 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/708272f4-3759-4997-a4b1-e36e1446410a_calibrate_1123955_25.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/708272f4-3759-4997-a4b1-e36e1446410a_calibrate_1123955_25.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f2c52014-bda5-47e7-a7b7-92b252872e09_characterizeImage_1123955_25.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f2c52014-bda5-47e7-a7b7-92b252872e09_characterizeImage_1123955_25.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4de5fe051e17886edb4167539ec29d15 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 8b588ba0c2d20b23b2901d4895f1ae3a parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639607c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3a358e1b-dbbf-4426-80cb-062b0fbacc17_calibrate_1123955_22.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3a358e1b-dbbf-4426-80cb-062b0fbacc17_calibrate_1123955_22.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/41acb19b-9da1-4752-8925-0cb41b0574d4_characterizeImage_1123955_22.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/41acb19b-9da1-4752-8925-0cb41b0574d4_characterizeImage_1123955_22.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b5d9d321facba64ac4ef6faeb291ac with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 7b05b9abc31c4f66349c4f6420f45683 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639afd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d9967d76-e397-468c-b766-214e160c8d3b_calibrate_1123955_45.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d9967d76-e397-468c-b766-214e160c8d3b_calibrate_1123955_45.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3a86d2d8-b169-4c8c-a785-b21a47dbd2fc_characterizeImage_1123955_45.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/3a86d2d8-b169-4c8c-a785-b21a47dbd2fc_characterizeImage_1123955_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 640 try 0 failed parsl.dataflow.dflow INFO: Task 640 marked for retry parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash bf7a06ed9eb0a5b3afd1cd25772b0434 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 1 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2e636690f5093be301a212179109e78 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 98ace61dd03d6a2e3878216f3a8358a8 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63905800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b0fbdf4f-e61b-4619-8367-808aa0c6706c_calibrate_1123957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b0fbdf4f-e61b-4619-8367-808aa0c6706c_calibrate_1123957_53.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe0cfc1-802d-4465-a85f-6a51b887c6e2_characterizeImage_1123957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bbe0cfc1-802d-4465-a85f-6a51b887c6e2_characterizeImage_1123957_53.stderr 2024-03-07 00:42:19 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:42:19 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7b344dff6bdb8ddb854c8312c5e7d46 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 9cd6908cc5d2dec3d94d1717328d9766 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f73fa3-63a3-4194-91a1-c26d33838dd9_calibrate_1123955_37.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f73fa3-63a3-4194-91a1-c26d33838dd9_calibrate_1123955_37.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/6a997b25-bd59-4c3f-a27c-94cad633d996_characterizeImage_1123955_37.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/6a997b25-bd59-4c3f-a27c-94cad633d996_characterizeImage_1123955_37.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 691 try 0 failed parsl.dataflow.dflow INFO: Task 691 marked for retry parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash be24cf2f69351ea2ea79d0cb66c19d43 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 1 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:21 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:22 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:23 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:23 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:24 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 707893ede8ac46c96281bb47d3b9d1d5 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash bcd8eb22aa5d63933c9eb15045819510 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c7573631-353b-406e-b2fa-4d430edade10_calibrate_1123957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c7573631-353b-406e-b2fa-4d430edade10_calibrate_1123957_18.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a5779976-d425-4b1c-99fb-94ddf32b7047_characterizeImage_1123957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/a5779976-d425-4b1c-99fb-94ddf32b7047_characterizeImage_1123957_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:24 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:24 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcd612b1806e3ab0b7a24345a12becdf with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 6053c942044850a96282a7047e48a284 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bfff94f0-be20-447d-9021-c3c3c4debe65_calibrate_1123953_34.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bfff94f0-be20-447d-9021-c3c3c4debe65_calibrate_1123953_34.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ff6ba8b3-820f-4b18-b032-6b40c251ccec_characterizeImage_1123953_34.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ff6ba8b3-820f-4b18-b032-6b40c251ccec_characterizeImage_1123953_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:25 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe2f5d70b6fb69f08693249dc78b057 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 515b7f71b1657df8daca67be275de806 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/43b1924c-2846-486d-b484-5213f89b0a21_calibrate_1123957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/43b1924c-2846-486d-b484-5213f89b0a21_calibrate_1123957_45.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/89797aa6-4662-4a0a-a0f4-f8758c19fb3c_characterizeImage_1123957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/89797aa6-4662-4a0a-a0f4-f8758c19fb3c_characterizeImage_1123957_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:25 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 1 failed parsl.dataflow.dflow ERROR: Task 4 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 44338a6df7e2e5c5fe5e16be33a4e281 with result from task 4 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 5 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 5 try 0 failed parsl.dataflow.dflow INFO: Task 5 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/2dc34453-8899-48d1-a41e-e57212f1f057_calibrate_1124086_58.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/2dc34453-8899-48d1-a41e-e57212f1f057_calibrate_1124086_58.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a46ce75-5324-4dfe-b203-e3f5c2fc9114_characterizeImage_1124086_58.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8afb08df356ec4084e39749eb4a1f3ad with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash d6e731e491527e9cb9742c412e9c8bf9 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c77a9054-4b2e-4584-8c39-fd3958337811_calibrate_1123957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c77a9054-4b2e-4584-8c39-fd3958337811_calibrate_1123957_25.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4aae4d14-bd14-459a-b465-8c5a484dc5ce_characterizeImage_1123957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4aae4d14-bd14-459a-b465-8c5a484dc5ce_characterizeImage_1123957_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:25 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1818d97f6f9ed034688676c0de8e94c6 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash f9c3f8af11c0e52018d7ad1c33b23eb8 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63963060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/33f890d2-ecd8-4be5-af7b-0d6626138010_calibrate_1123957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/33f890d2-ecd8-4be5-af7b-0d6626138010_calibrate_1123957_48.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a1a02e5-451e-428b-817a-f6a82123ed9a_characterizeImage_1123957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a1a02e5-451e-428b-817a-f6a82123ed9a_characterizeImage_1123957_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:26 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:26 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:26 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:27 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:27 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:28 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:28 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:28 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e68f2ee4405bcd41514d3417ddd162f5 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 6069a93b575e2134092d77ad224e7c6e parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5f91d19b-2c7a-4baf-b089-d54b238dcf5e_calibrate_1123955_62.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5f91d19b-2c7a-4baf-b089-d54b238dcf5e_calibrate_1123955_62.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5400c5d-ad98-4948-8700-3008e99a1a0e_characterizeImage_1123955_62.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5400c5d-ad98-4948-8700-3008e99a1a0e_characterizeImage_1123955_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:29 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:29 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:29 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:30 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:30 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:30 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:31 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 673 try 0 failed parsl.dataflow.dflow INFO: Task 673 marked for retry parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 48b5a3340b4cbb113a64d14de752a40b parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 1 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:31 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 199 try 1 failed parsl.dataflow.dflow ERROR: Task 199 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f2d6e8b33b62550cd6ca9feefa25bf06 with result from task 199 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 200 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 200 try 0 failed parsl.dataflow.dflow INFO: Task 200 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/99986311-a51c-4824-8013-5222ab7c8b73_calibrate_1123955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/99986311-a51c-4824-8013-5222ab7c8b73_calibrate_1123955_2.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d283ef3b-f115-4cf2-8bb3-0c3db426fa61_characterizeImage_1123955_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:31 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:32 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a157bf2c93e31a3d1d0d71bab98c0079 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash d2491803816ae949c44c7659679e466e parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628082c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0b83a737-5455-4bbb-be00-d120d5a4bc3f_calibrate_1123957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0b83a737-5455-4bbb-be00-d120d5a4bc3f_calibrate_1123957_5.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/360ed409-b291-4454-8c50-46802605073d_characterizeImage_1123957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/360ed409-b291-4454-8c50-46802605073d_characterizeImage_1123957_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:32 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:32 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:33 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:33 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:33 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 289 try 1 failed parsl.dataflow.dflow ERROR: Task 289 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 57204c40005aedc45d3730f350fd0a55 with result from task 289 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 290 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 290 try 0 failed parsl.dataflow.dflow INFO: Task 290 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d0230a2d-3e3d-455a-a9e1-4055b20b4293_calibrate_1124086_5.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d0230a2d-3e3d-455a-a9e1-4055b20b4293_calibrate_1124086_5.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/804fe993-6469-4fed-ba25-87acf2187e6d_characterizeImage_1124086_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:34 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 715 try 0 failed parsl.dataflow.dflow INFO: Task 715 marked for retry parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 4b2c0b62908ef6c5e468082bb994e87b parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 1 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:34 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:34 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:34 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:35 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:35 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c968a3db1002cd4529ece187dbae7f7 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 79b54366a22fcbfbda1d9bab16386963 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3e97db99-93e9-4b38-bb07-3abf4cc54e44_calibrate_1123957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3e97db99-93e9-4b38-bb07-3abf4cc54e44_calibrate_1123957_14.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/954084de-2208-46bc-a3e4-82b61949ba05_characterizeImage_1123957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/954084de-2208-46bc-a3e4-82b61949ba05_characterizeImage_1123957_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:35 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 151 try 0 failed parsl.dataflow.dflow INFO: Task 151 marked for retry parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 8a79d7bf52e2cbb96762550b3eb969f7 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:36 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Parsl task 151 try 1 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:36 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 73 try 1 failed parsl.dataflow.dflow ERROR: Task 73 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 804ae82d5ae294d546a262a0813a41fa with result from task 73 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 74 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 74 try 0 failed parsl.dataflow.dflow INFO: Task 74 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:36 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/39d090ed-056b-43d2-a23a-1393991092ab_calibrate_1124086_47.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/39d090ed-056b-43d2-a23a-1393991092ab_calibrate_1124086_47.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7e6bacd7-5e58-4aea-8736-04b1e14c557f_characterizeImage_1124086_47.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01503387b5ab9b5094ae0c060565ca2c with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 1ec5f854783a7b82debbeb0ba31e3f03 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/61390964-40c1-4469-80ec-c49e39b716f9_calibrate_1123957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/61390964-40c1-4469-80ec-c49e39b716f9_calibrate_1123957_1.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0d79ead7-d33d-4717-95a2-01cd7bbe9c92_characterizeImage_1123957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0d79ead7-d33d-4717-95a2-01cd7bbe9c92_characterizeImage_1123957_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:37 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:37 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:37 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 667 try 0 failed parsl.dataflow.dflow INFO: Task 667 marked for retry parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 3ae032be8cd92868527afb26acdf35ec parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 1 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 1 failed parsl.dataflow.dflow ERROR: Task 100 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1f7b5a60331c086cf06ceed9750c6504 with result from task 100 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 101 try 0 failed parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4f3ccee4-2e31-48f1-9b28-3b8200f3900b_calibrate_1124086_9.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4f3ccee4-2e31-48f1-9b28-3b8200f3900b_calibrate_1124086_9.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d91c08f6-9f9f-46a0-b619-c5287e32c38b_characterizeImage_1124086_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:38 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:38 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:38 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30ac0dc9cef3a5944923cc16a1da9084 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 0f8c6f6d1562416fa7d3150f82b610bd parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/9efcf8ee-565a-4790-9d6a-28c2fa160061_calibrate_1123953_57.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/9efcf8ee-565a-4790-9d6a-28c2fa160061_calibrate_1123953_57.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8f113fa1-91b5-4ae0-8477-0acdbe840a32_characterizeImage_1123953_57.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/8f113fa1-91b5-4ae0-8477-0acdbe840a32_characterizeImage_1123953_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:39 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:39 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:39 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:40 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56ed1e3c7d36d2aa67096258be0225e9 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 7189a504755be783506fb510cc21f8d0 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b66b485f-e5ff-4c00-bce6-776669f54473_calibrate_1123955_16.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b66b485f-e5ff-4c00-bce6-776669f54473_calibrate_1123955_16.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/7145d158-c07e-4cac-a408-9060bff07cc5_characterizeImage_1123955_16.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/7145d158-c07e-4cac-a408-9060bff07cc5_characterizeImage_1123955_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:40 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:41 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8448a8bb24bfa6eeee15e183b2e321d0 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 157a7e1bb8446b1babd71efe2e15d9ac parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/74387827-6623-4e77-a828-a290e7f4f998_calibrate_1123957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/74387827-6623-4e77-a828-a290e7f4f998_calibrate_1123957_34.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1e2c2e78-85e1-4789-a56a-c7cab4789362_characterizeImage_1123957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1e2c2e78-85e1-4789-a56a-c7cab4789362_characterizeImage_1123957_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:41 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 1 failed parsl.dataflow.dflow ERROR: Task 1 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2424cd0973a1a05fd83524ac06b799cc with result from task 1 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 2 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 2 try 0 failed parsl.dataflow.dflow INFO: Task 2 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/c349e2c7-857f-4bfc-adf9-baf137a5a376_calibrate_1124086_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/c349e2c7-857f-4bfc-adf9-baf137a5a376_calibrate_1124086_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d1ec1c91-84bc-48bf-9518-ae597cf370ab_characterizeImage_1124086_16.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b47d990e69026e78dde2ce4f4274d2d6 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash a652c93b9f7cebe6812bbfe3d14d4492 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7f488037-0ddd-4048-9ae8-4896b5ccc5c4_calibrate_1123955_57.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7f488037-0ddd-4048-9ae8-4896b5ccc5c4_calibrate_1123955_57.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/68bcce6d-3b45-42b5-9cd7-3ecfae94d229_characterizeImage_1123955_57.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/68bcce6d-3b45-42b5-9cd7-3ecfae94d229_characterizeImage_1123955_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:41 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:42 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62afcf3290f2c5b0c108296b415ac856 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 53e85594ad063e8b5b4676018898a066 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f1ae02ce-080b-4167-853b-ac928223ef69_calibrate_1123953_54.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f1ae02ce-080b-4167-853b-ac928223ef69_calibrate_1123953_54.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a6040370-0f34-477e-ba11-ba02ed223e0f_characterizeImage_1123953_54.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a6040370-0f34-477e-ba11-ba02ed223e0f_characterizeImage_1123953_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:42 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:42 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13abd2443d18d526c6a0fa200294137d with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash c9f3422d4ac8558de81133455885c4e8 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639dbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/fe40d186-f8c4-4a4e-b342-6aa559f39243_calibrate_1123953_35.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/fe40d186-f8c4-4a4e-b342-6aa559f39243_calibrate_1123953_35.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2f7f86d5-554e-43fe-94e8-5ddb7d6317c5_characterizeImage_1123953_35.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2f7f86d5-554e-43fe-94e8-5ddb7d6317c5_characterizeImage_1123953_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 R 16882982 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 242 parsl.dataflow.strategy DEBUG: Executor multi has 242 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 242 parsl.dataflow.strategy DEBUG: Executor multi has 242 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 1 failed parsl.dataflow.dflow ERROR: Task 22 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ca5e1a1ee94b8e173b279fbcd3986adf with result from task 22 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 23 try 0 failed parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/94857a15-cdaf-483c-9e6a-4488d8f03cb2_calibrate_1124086_51.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/94857a15-cdaf-483c-9e6a-4488d8f03cb2_calibrate_1124086_51.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ccb6ca18-f987-4447-a753-52f9fb89f7ee_characterizeImage_1124086_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 322 try 1 failed parsl.dataflow.dflow ERROR: Task 322 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a1b0f9a2d8c46eb08134af70849f44b1 with result from task 322 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 323 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 323 try 0 failed parsl.dataflow.dflow INFO: Task 323 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/7e792e3a-26c3-4a48-92ce-72e1bc15ba7f_calibrate_1124086_10.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/7e792e3a-26c3-4a48-92ce-72e1bc15ba7f_calibrate_1124086_10.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5202316d-480f-4dd7-9d25-a48b011cf708_characterizeImage_1124086_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 79 try 1 failed parsl.dataflow.dflow ERROR: Task 79 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 66a5c7d65ac0913a274af22ce100e4c6 with result from task 79 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 80 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 80 try 0 failed parsl.dataflow.dflow INFO: Task 80 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/cf21a3a2-ef7a-4b7e-93fd-1e50e9cdca1d_calibrate_1124086_44.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/cf21a3a2-ef7a-4b7e-93fd-1e50e9cdca1d_calibrate_1124086_44.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e9bf7bcd-3a2a-419a-8c7c-1a0e2027608b_characterizeImage_1124086_44.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa335fa859d484b9cb909b927c97d81b with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 0695596058db808e60d9099d0dbaa292 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/84bce3ea-bddb-44bb-807e-3fc408487232_calibrate_1123957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/84bce3ea-bddb-44bb-807e-3fc408487232_calibrate_1123957_56.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f4dfc7a6-8392-42ee-aa36-7c6267d159ac_characterizeImage_1123957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f4dfc7a6-8392-42ee-aa36-7c6267d159ac_characterizeImage_1123957_56.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 109 try 1 failed parsl.dataflow.dflow ERROR: Task 109 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b34a0bb75ddd9d6447b8887a9393d9ba with result from task 109 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 110 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 110 try 0 failed parsl.dataflow.dflow INFO: Task 110 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/5b2a69c1-c131-400a-a331-03ba3788c15e_calibrate_1124086_57.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/5b2a69c1-c131-400a-a331-03ba3788c15e_calibrate_1124086_57.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f3448a24-89e8-47e4-a70f-ac15d9abafa6_characterizeImage_1124086_57.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 31 try 1 failed parsl.dataflow.dflow ERROR: Task 31 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fd5e60ca0499ebf65b2043b1bb9c1455 with result from task 31 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 32 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 32 try 0 failed parsl.dataflow.dflow INFO: Task 32 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/a39a3e31-84f9-4151-a369-3859c2ee6054_calibrate_1124086_19.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/a39a3e31-84f9-4151-a369-3859c2ee6054_calibrate_1124086_19.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/2b15b533-5a17-43a2-95b6-b11954f2cf63_characterizeImage_1124086_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 1 failed parsl.dataflow.dflow ERROR: Task 52 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2e17cc24ee39e5b65e2951f1cd5142dd with result from task 52 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 53 try 0 failed parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/efc1ae73-6f45-4502-8293-e52be5e133d4_calibrate_1124086_62.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/efc1ae73-6f45-4502-8293-e52be5e133d4_calibrate_1124086_62.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/3c9eb0d2-90dd-49b5-9fbc-491d6a93dba6_characterizeImage_1124086_62.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56ace1f57ff683bfa0511c6d6b5e0dd4 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash a947ab5ec9c0d4280583a19a7751c097 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62899300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/54682175-cfdb-43a8-ab9c-adb3a328b0ed_calibrate_1123953_26.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/54682175-cfdb-43a8-ab9c-adb3a328b0ed_calibrate_1123953_26.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/5ac0beef-a1fc-40ff-87ce-216c4bc181b8_characterizeImage_1123953_26.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/5ac0beef-a1fc-40ff-87ce-216c4bc181b8_characterizeImage_1123953_26.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 919f2ac9cdda7f74305711c309b332ca with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 8472ea7e99e054b486ab560b47ac5d24 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/327b1949-6f9a-49e6-932d-15c7f90f3331_calibrate_1123957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/327b1949-6f9a-49e6-932d-15c7f90f3331_calibrate_1123957_32.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/05b13db4-a38a-461f-b7dd-1e8eddd33a71_characterizeImage_1123957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/05b13db4-a38a-461f-b7dd-1e8eddd33a71_characterizeImage_1123957_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 277 try 1 failed parsl.dataflow.dflow ERROR: Task 277 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry da4ced1143be1785788d565253a703af with result from task 277 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 278 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 278 try 0 failed parsl.dataflow.dflow INFO: Task 278 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/df10d81f-c8d8-4525-b982-d302994ef515_calibrate_1124086_28.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/df10d81f-c8d8-4525-b982-d302994ef515_calibrate_1124086_28.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/73923b66-1802-49ee-a09c-8e6bc011738b_characterizeImage_1124086_28.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8d42c7cfd32e556456ab2a7c7b879f5 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash fee5efa37c1286fd80aa46b372aa412c parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a4a41427-24a4-4718-ae72-9eeaabba067e_calibrate_1123955_9.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a4a41427-24a4-4718-ae72-9eeaabba067e_calibrate_1123955_9.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5fd72b72-5e17-4a7f-a286-00a19ef9328c_characterizeImage_1123955_9.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5fd72b72-5e17-4a7f-a286-00a19ef9328c_characterizeImage_1123955_9.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d67266c64715abf95f829cedfcff852b with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 8988013613219bd930132cf987bf8c68 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e0d2d8e8-e8bf-4d8b-b1e4-9ec6baceba68_calibrate_1123953_14.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e0d2d8e8-e8bf-4d8b-b1e4-9ec6baceba68_calibrate_1123953_14.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/12065c99-f0bc-4aab-b178-0b90a67a2e9a_characterizeImage_1123953_14.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/12065c99-f0bc-4aab-b178-0b90a67a2e9a_characterizeImage_1123953_14.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0668936f9e000eab8905cac2b346d94 with result from task 481 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 19db1fc17d0a805d4e3275d28a5cfeea parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3313c30a-e3fb-43bb-aa3a-3851757022fa_calibrate_1123953_16.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3313c30a-e3fb-43bb-aa3a-3851757022fa_calibrate_1123953_16.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/659cb27b-1950-41df-aa27-82ec8c888d49_characterizeImage_1123953_16.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/659cb27b-1950-41df-aa27-82ec8c888d49_characterizeImage_1123953_16.stderr parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 234 parsl.dataflow.strategy DEBUG: Executor multi has 234 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9203958242ace9238835f884f77597a6 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash a18117defb7b018436ab0d6d0d3e0724 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/871e00c6-1e8f-4390-9c86-6935443824f4_calibrate_1123953_32.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/871e00c6-1e8f-4390-9c86-6935443824f4_calibrate_1123953_32.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/681ae588-d7d7-4427-9016-88cdf6b26534_characterizeImage_1123953_32.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/681ae588-d7d7-4427-9016-88cdf6b26534_characterizeImage_1123953_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 169 try 1 failed parsl.dataflow.dflow ERROR: Task 169 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 570769d5c76ec82bcac0109a091b40ca with result from task 169 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 170 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 170 try 0 failed parsl.dataflow.dflow INFO: Task 170 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/13dfc191-a79b-4ffb-9f69-a5ee637c42ad_calibrate_1124086_54.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/13dfc191-a79b-4ffb-9f69-a5ee637c42ad_calibrate_1124086_54.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/25ce7cdf-e1fe-4d71-a076-7ca8a29575d6_characterizeImage_1124086_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 184 try 1 failed parsl.dataflow.dflow ERROR: Task 184 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry af8f3b018eb2730835430d25b15b9630 with result from task 184 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 185 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 185 try 0 failed parsl.dataflow.dflow INFO: Task 185 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/8e018f59-c8e6-45fd-9d83-f9aee4d9100f_calibrate_1124086_8.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/8e018f59-c8e6-45fd-9d83-f9aee4d9100f_calibrate_1124086_8.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/6f31a5d0-5f75-4453-bdba-a4d92e8a8fea_characterizeImage_1124086_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 253 try 1 failed parsl.dataflow.dflow ERROR: Task 253 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 24c76bf06b06d647d038149601c67b11 with result from task 253 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 254 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 254 try 0 failed parsl.dataflow.dflow INFO: Task 254 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/ab87eb0e-1068-480a-9fc6-2d5149097331_calibrate_1124086_20.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/ab87eb0e-1068-480a-9fc6-2d5149097331_calibrate_1124086_20.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1d94a513-4fa2-4fc5-9894-cb2f9da90770_characterizeImage_1124086_20.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1aee431e08fa8cbe405271ac20575a with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 14e10ac81d9b0e5d822cf2d1b355a62a parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62808900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6475e2ef-ad1c-402b-8b8c-7a0d7bedea45_calibrate_1123953_41.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6475e2ef-ad1c-402b-8b8c-7a0d7bedea45_calibrate_1123953_41.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a52f4ec6-7f6a-4952-a896-0fcb926bdd75_characterizeImage_1123953_41.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/a52f4ec6-7f6a-4952-a896-0fcb926bdd75_characterizeImage_1123953_41.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b44cb50f33a067e40b5b8fd97ea7fdf3 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash cc705b6a746c32471c8f363ab1bb8b26 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc6536b2-d843-42e1-8643-5ef30294e03d_calibrate_1123953_52.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc6536b2-d843-42e1-8643-5ef30294e03d_calibrate_1123953_52.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/91abe08d-c6aa-47fa-b7fe-839aaddb7be5_characterizeImage_1123953_52.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/91abe08d-c6aa-47fa-b7fe-839aaddb7be5_characterizeImage_1123953_52.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e5de02eab2fe0f0792084814768a1b4 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash b4d2d0261a37b94b239dc235b86e5af4 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3d31e96e-9914-4fd3-bd9c-a4c7d19a77ac_calibrate_1123957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3d31e96e-9914-4fd3-bd9c-a4c7d19a77ac_calibrate_1123957_35.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/60a1afe4-8a2f-4b2f-ad78-6bbbdd50cdce_characterizeImage_1123957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/60a1afe4-8a2f-4b2f-ad78-6bbbdd50cdce_characterizeImage_1123957_35.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 346 try 0 failed parsl.dataflow.dflow INFO: Task 346 marked for retry parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 1d8308824eaf28aad553a8cef558f49b parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 1 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stderr 2024-03-07 00:42:53 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 361 try 0 failed parsl.dataflow.dflow INFO: Task 361 marked for retry parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash a2539746568cd4d0c7e943d5339c97eb parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 1 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stderr 2024-03-07 00:42:53 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:53 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 307 try 1 failed parsl.dataflow.dflow ERROR: Task 307 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 97ece9fc32bb40738a50be9516ad7430 with result from task 307 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 308 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 308 try 0 failed parsl.dataflow.dflow INFO: Task 308 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/13b6366c-3b4a-414c-9b77-4929251c3562_calibrate_1124086_32.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/13b6366c-3b4a-414c-9b77-4929251c3562_calibrate_1124086_32.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/20686030-878e-4dd5-a239-f8f09da0acd6_characterizeImage_1124086_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 97 try 1 failed parsl.dataflow.dflow ERROR: Task 97 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f09886a46d581e9370265c46f1e5b8c1 with result from task 97 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 98 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 98 try 0 failed parsl.dataflow.dflow INFO: Task 98 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/12283d97-ba8a-43e6-988f-6b49226bfa04_calibrate_1124086_48.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/12283d97-ba8a-43e6-988f-6b49226bfa04_calibrate_1124086_48.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/89ea6790-80e0-4993-84b1-3baec837a789_characterizeImage_1124086_48.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 1 failed parsl.dataflow.dflow ERROR: Task 118 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2d022c45144f0d07758316012d546f8a with result from task 118 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 119 try 0 failed parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/b82db67f-12aa-4e6e-b815-574160f9c127_calibrate_1124086_49.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/b82db67f-12aa-4e6e-b815-574160f9c127_calibrate_1124086_49.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/365fc5f0-bab0-4530-9804-06537789a202_characterizeImage_1124086_49.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 235 try 1 failed parsl.dataflow.dflow ERROR: Task 235 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dd9d58b0a42d6eb58f8f214eafbfeac7 with result from task 235 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 236 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 236 try 0 failed parsl.dataflow.dflow INFO: Task 236 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/6d95152c-fed2-44d7-bded-338dce7ce8a6_calibrate_1124086_6.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/6d95152c-fed2-44d7-bded-338dce7ce8a6_calibrate_1124086_6.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f57cda25-0144-4454-ac4f-dc9b55c9a56c_characterizeImage_1124086_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 655 try 0 failed parsl.dataflow.dflow INFO: Task 655 marked for retry parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash a8c23f316931b637a7912e16787a011f parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 1 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c76b1bfb79aac8150ec88ae347e1d79 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash d054d10815c5bfd8006cf201038b2a3b parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639636a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e016df84-d5fa-45a1-975e-2d7b192a5f1f_calibrate_1123953_29.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e016df84-d5fa-45a1-975e-2d7b192a5f1f_calibrate_1123953_29.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb7a67a4-f503-4240-b893-fedf5fa8e9b5_characterizeImage_1123953_29.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb7a67a4-f503-4240-b893-fedf5fa8e9b5_characterizeImage_1123953_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:55 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adcc4104c627a086e93f8326e9d728f3 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 635feb60cc29de8a31386bf9308382b7 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/45493781-592a-49a4-9991-2120f57aef8d_calibrate_1123953_11.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/45493781-592a-49a4-9991-2120f57aef8d_calibrate_1123953_11.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/784be170-f29f-4282-8b76-0334bb12e0a5_characterizeImage_1123953_11.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/784be170-f29f-4282-8b76-0334bb12e0a5_characterizeImage_1123953_11.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 367 try 0 failed parsl.dataflow.dflow INFO: Task 367 marked for retry parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash c26aa2901edd56434605a1b53a2f724f parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 1 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:55 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 376 try 1 failed parsl.dataflow.dflow ERROR: Task 376 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry acf2b064af358e67396ef9f14da38cac with result from task 376 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 377 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 377 try 0 failed parsl.dataflow.dflow INFO: Task 377 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/eebdf4b3-02ee-4017-96f9-494ceb7da231_calibrate_1124086_7.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/eebdf4b3-02ee-4017-96f9-494ceb7da231_calibrate_1124086_7.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f066357b-7a27-4220-9193-562a5c0be25f_characterizeImage_1124086_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 214 try 1 failed parsl.dataflow.dflow ERROR: Task 214 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a704ab4d1dd1738c9d5fd0e11ca02d64 with result from task 214 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 215 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 215 try 0 failed parsl.dataflow.dflow INFO: Task 215 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/13a6b1ce-4e0d-46d7-96eb-d268e96dd8a0_calibrate_1124086_53.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/13a6b1ce-4e0d-46d7-96eb-d268e96dd8a0_calibrate_1124086_53.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/52323962-3496-49fa-ac73-0263b45fe9f4_characterizeImage_1124086_53.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec781539d66d719ad40bb6e784d5019f with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 49797cfd98ae0bced97fb9e47b72b2c8 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/41f76e45-6ccb-44d0-893b-afef88025748_calibrate_1123955_29.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/41f76e45-6ccb-44d0-893b-afef88025748_calibrate_1123955_29.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d1037579-f28d-4b6e-9e90-b3a786a5e4ee_characterizeImage_1123955_29.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d1037579-f28d-4b6e-9e90-b3a786a5e4ee_characterizeImage_1123955_29.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 421 try 1 failed parsl.dataflow.dflow ERROR: Task 421 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9d330e4c97e2b8d9486e21f7e23e1c68 with result from task 421 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 422 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 422 try 0 failed parsl.dataflow.dflow INFO: Task 422 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f3f98718-f90e-4b8d-b806-b0dc5889b2a7_calibrate_1123953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f3f98718-f90e-4b8d-b806-b0dc5889b2a7_calibrate_1123953_2.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bb90f05b-b078-437b-8533-c0d50b57d96c_characterizeImage_1123953_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:57 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7548cefae1f1649d3075d0a093f2fc46 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash cedfdfe82434902774ba45c4f6f192e5 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3bbc640a-6189-4590-a02c-d1eca55083df_calibrate_1123955_1.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3bbc640a-6189-4590-a02c-d1eca55083df_calibrate_1123955_1.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bce4d9-0185-4ef7-8f0d-309f1f63f081_characterizeImage_1123955_1.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/91bce4d9-0185-4ef7-8f0d-309f1f63f081_characterizeImage_1123955_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:57 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:58 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:58 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:58 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22a6ec1a6950101db4b64c45b65820bf with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash fd74386b5a469c21343405e10900c8ec parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/73c475c4-75f3-4e97-98dd-7d364ecfe23b_calibrate_1123957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/73c475c4-75f3-4e97-98dd-7d364ecfe23b_calibrate_1123957_27.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/2cef785f-d0f8-4973-a5dc-6cb0523b41a9_characterizeImage_1123957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/2cef785f-d0f8-4973-a5dc-6cb0523b41a9_characterizeImage_1123957_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:59 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:59 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:59 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:59 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0279f28fc5518425b4b48e3de838209c with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 2bf6d1141d6adb6a88b95d953aeb0a01 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fb7cc54f-be81-4200-82fe-e76211a3e9bd_calibrate_1123955_35.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fb7cc54f-be81-4200-82fe-e76211a3e9bd_calibrate_1123955_35.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ee763f54-4805-428c-a643-9e67752341b8_characterizeImage_1123955_35.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ee763f54-4805-428c-a643-9e67752341b8_characterizeImage_1123955_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:42:59 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:00 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 280 try 1 failed parsl.dataflow.dflow ERROR: Task 280 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7b6ff2c8ebaa5d2bb20c7fa2e2fb1a4b with result from task 280 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 281 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 281 try 0 failed parsl.dataflow.dflow INFO: Task 281 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/0ec24d96-acb8-4d52-9eb1-5a906ae8cf17_calibrate_1124086_38.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/0ec24d96-acb8-4d52-9eb1-5a906ae8cf17_calibrate_1124086_38.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/96573cf3-624c-431c-9e3b-254277ec8bf0_characterizeImage_1124086_38.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d08d6cf00659c1696c221fba9d0ce59 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 674e9668989ec2df423d68686ffa3a02 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/47a4a806-c7d8-49ca-bf9d-ed45a852a749_calibrate_1123953_51.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/47a4a806-c7d8-49ca-bf9d-ed45a852a749_calibrate_1123953_51.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e0619304-9f6f-44a4-ae05-0d549ec5844d_characterizeImage_1123953_51.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e0619304-9f6f-44a4-ae05-0d549ec5844d_characterizeImage_1123953_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:00 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f1d18262ef1c5363773431dd52f2d6d with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash f8e26ad24ed407b8439d4d2736a81bcb parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/09d507d9-a78d-414c-8124-5926cf79f1cb_calibrate_1123953_10.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/09d507d9-a78d-414c-8124-5926cf79f1cb_calibrate_1123953_10.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b4dd492b-3090-495d-b565-3645fd9f6dc8_characterizeImage_1123953_10.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b4dd492b-3090-495d-b565-3645fd9f6dc8_characterizeImage_1123953_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:01 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29ed57fc1f01d9bb43511ecbf6301c8e with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 0dea23dea5e470be92b04e427840037f parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7017832d-1ff5-4f42-8f4d-e5e0c460bc60_calibrate_1123957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7017832d-1ff5-4f42-8f4d-e5e0c460bc60_calibrate_1123957_11.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ed676976-b865-4896-8e93-d90d916fac31_characterizeImage_1123957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ed676976-b865-4896-8e93-d90d916fac31_characterizeImage_1123957_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:01 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:01 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 153d2a1537d55c51f0e27ecfd575f771 with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 01bc92a6916d3997cef87d078425db88 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/291ac453-1482-48f6-95ea-2d993bce613d_calibrate_1123957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/291ac453-1482-48f6-95ea-2d993bce613d_calibrate_1123957_36.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/669aba42-d642-4505-86de-d33ba45963d4_characterizeImage_1123957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/669aba42-d642-4505-86de-d33ba45963d4_characterizeImage_1123957_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:02 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:02 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 340ccf5ad7f7fcd63fbdcbe91c6297d6 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 1cb3041605c2dbde93c7084a98fe39f3 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/be839559-212a-4b61-b3b0-0ddec2667a09_calibrate_1123957_8.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/be839559-212a-4b61-b3b0-0ddec2667a09_calibrate_1123957_8.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/685bd1c3-37b5-45ef-85bc-2f707b6bd544_characterizeImage_1123957_8.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/685bd1c3-37b5-45ef-85bc-2f707b6bd544_characterizeImage_1123957_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:02 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:03 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f25891e6c1f71b8133fa05832d433c7c with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 53500d9b41450f6303fd9cd15b0ff6b4 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6d840c1e-0e3f-4abc-82cc-a3a0482f9fef_calibrate_1123953_15.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6d840c1e-0e3f-4abc-82cc-a3a0482f9fef_calibrate_1123953_15.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/84dd5a21-e911-46ca-960b-ca3cce372eb8_characterizeImage_1123953_15.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/84dd5a21-e911-46ca-960b-ca3cce372eb8_characterizeImage_1123953_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:03 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32e6715dcddae07fca8bfde23297880f with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash c3c377554d80220dea477219e0452c08 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7011f473-a877-4a0c-a1c3-b4364c304be9_calibrate_1123955_30.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7011f473-a877-4a0c-a1c3-b4364c304be9_calibrate_1123955_30.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8ca57102-0929-4172-aa6d-d74d2127065a_characterizeImage_1123955_30.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/8ca57102-0929-4172-aa6d-d74d2127065a_characterizeImage_1123955_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:04 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:04 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:04 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7acc791ed6d365c80b4ff45f278be3e with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash cf40222348efc767323285e14bc484cb parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4f87ca71-d52b-4b32-bd99-4d6fa22442c5_calibrate_1123953_27.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4f87ca71-d52b-4b32-bd99-4d6fa22442c5_calibrate_1123953_27.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7d970114-a16b-46b1-9172-200e678a0f10_characterizeImage_1123953_27.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7d970114-a16b-46b1-9172-200e678a0f10_characterizeImage_1123953_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:05 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:05 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:05 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:06 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 232 try 1 failed parsl.dataflow.dflow ERROR: Task 232 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4468ec30f1497c7fc5358bf13609d8ed with result from task 232 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 233 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 233 try 0 failed parsl.dataflow.dflow INFO: Task 233 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/926ba7f7-d37a-4720-a324-3924e599be4c_calibrate_1124086_12.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/926ba7f7-d37a-4720-a324-3924e599be4c_calibrate_1124086_12.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/4575e873-f844-4532-ad75-e116b20ac747_characterizeImage_1124086_12.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 295 try 1 failed parsl.dataflow.dflow ERROR: Task 295 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 977948b6d97373167b64a58a7610d16c with result from task 295 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 296 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 296 try 0 failed parsl.dataflow.dflow INFO: Task 296 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/53781f80-9c4a-4510-973f-e8a7a4de0dbc_calibrate_1124086_30.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/53781f80-9c4a-4510-973f-e8a7a4de0dbc_calibrate_1124086_30.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/1a0460a9-6e57-4466-9eb3-868835e93c3b_characterizeImage_1124086_30.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a82209d9a47057f17137adc82091cb4 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 0344ab7a6b29e7958ad2aa196db309cb parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0fa0cd5d-9289-40fe-a437-72e277534010_calibrate_1123953_45.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0fa0cd5d-9289-40fe-a437-72e277534010_calibrate_1123953_45.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25388e84-da08-446f-b945-deb3cc070e76_characterizeImage_1123953_45.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25388e84-da08-446f-b945-deb3cc070e76_characterizeImage_1123953_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:06 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:07 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e2788467f8d5fa379c288d9034741ae with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 0dcfaed311fb6fbaf5a90cc560557976 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bdf69a5a-f7c3-4479-bdab-67222f2cb1aa_calibrate_1123955_21.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bdf69a5a-f7c3-4479-bdab-67222f2cb1aa_calibrate_1123955_21.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2e675c9c-76be-44a9-95f4-fcae68e9c68d_characterizeImage_1123955_21.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2e675c9c-76be-44a9-95f4-fcae68e9c68d_characterizeImage_1123955_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:07 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:07 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 409 try 1 failed parsl.dataflow.dflow ERROR: Task 409 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1c731bfbed0ebde45e6ab1f1e4995d36 with result from task 409 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 410 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 410 try 0 failed parsl.dataflow.dflow INFO: Task 410 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/6be4b25a-058b-4f95-a3b5-bea846f9de7d_calibrate_1124086_46.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/6be4b25a-058b-4f95-a3b5-bea846f9de7d_calibrate_1124086_46.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34e0b3aa-b0fc-49eb-a1ac-440b5efc4dda_characterizeImage_1124086_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:08 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:08 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:08 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:09 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:09 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 385 try 1 failed parsl.dataflow.dflow ERROR: Task 385 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 23909fe7c63d1a66812555eb529fbcc4 with result from task 385 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 386 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 386 try 0 failed parsl.dataflow.dflow INFO: Task 386 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/401a4ed0-a5d1-4ada-b78a-d2328a1bbcda_calibrate_1124086_39.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/401a4ed0-a5d1-4ada-b78a-d2328a1bbcda_calibrate_1124086_39.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5133b644-a229-4423-9045-7eb9346dbc50_characterizeImage_1124086_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:10 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:10 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:10 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 547 try 1 failed parsl.dataflow.dflow ERROR: Task 547 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c28f4bf5b893737d3b073661120fb2ec with result from task 547 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 548 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 548 try 0 failed parsl.dataflow.dflow INFO: Task 548 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/34b45e29-3b27-46ad-9bde-35dd78a4b566_calibrate_1124086_25.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/34b45e29-3b27-46ad-9bde-35dd78a4b566_calibrate_1124086_25.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/102192de-b2e6-4018-867d-a7d2d9dea5c7_characterizeImage_1124086_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:11 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:11 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 694 try 0 failed parsl.dataflow.dflow INFO: Task 694 marked for retry parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 54c0a4e6f31cd52c053b2731c5493b68 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 1 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:12 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:12 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:12 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:13 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9af7d88fe2d1e9917c92faab15a58df with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 515696ec91d8367d5113e93b41b905b8 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63962d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/fdc8b17d-b77d-4fc9-b84d-a184b3ada5f0_calibrate_1123957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/fdc8b17d-b77d-4fc9-b84d-a184b3ada5f0_calibrate_1123957_4.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4a21411e-2c3f-4396-af4e-167a3ee77838_characterizeImage_1123957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4a21411e-2c3f-4396-af4e-167a3ee77838_characterizeImage_1123957_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:13 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:13 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:14 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b960543338037143fd8119ca3cf52c02 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 767c7177f762e448d293736ff9c18928 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5f3a6c1a-85b6-4bea-b714-0a3135a76734_calibrate_1123953_38.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5f3a6c1a-85b6-4bea-b714-0a3135a76734_calibrate_1123953_38.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e24fdb1f-87cd-4d5b-a76e-e6350d507cd0_characterizeImage_1123953_38.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e24fdb1f-87cd-4d5b-a76e-e6350d507cd0_characterizeImage_1123953_38.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3894354370e87de3d121415413292e56 with result from task 382 parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:14 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 3dedd728d5442c728ff8c8d966c3da3d parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5ec6c6f7-aad3-4da8-95e2-01ba2dbb9c60_calibrate_1123953_7.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5ec6c6f7-aad3-4da8-95e2-01ba2dbb9c60_calibrate_1123953_7.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/18fd6db9-6b6c-407d-9fc4-f37beb44c076_characterizeImage_1123953_7.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/18fd6db9-6b6c-407d-9fc4-f37beb44c076_characterizeImage_1123953_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:14 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:15 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:15 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 310 try 1 failed parsl.dataflow.dflow ERROR: Task 310 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a34ef020eff8f6f050f0786277892043 with result from task 310 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 311 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 311 try 0 failed parsl.dataflow.dflow INFO: Task 311 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/5380a786-b698-4692-bd7c-d7591cad14fb_calibrate_1124086_36.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/5380a786-b698-4692-bd7c-d7591cad14fb_calibrate_1124086_36.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5d93bacc-3c1b-48b1-aef0-a50f868675b4_characterizeImage_1124086_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 CG 16882983 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:16 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 CG 16882983 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:16 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 CG 16882983 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:16 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 568 try 1 failed parsl.dataflow.dflow ERROR: Task 568 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a62a0ec46601102648cc5a82724b1af0 with result from task 568 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 569 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 569 try 0 failed parsl.dataflow.dflow INFO: Task 569 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/0d65c87b-d153-4f68-b006-21c5580bae10_calibrate_1124086_4.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/0d65c87b-d153-4f68-b006-21c5580bae10_calibrate_1124086_4.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ee684167-90c2-485c-af08-e962660ba719_characterizeImage_1124086_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 CG 16882983 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:17 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 CG 16882983 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:17 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8a56a7bf7e517fd386f8e12df72befd with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash a11bd92b2874a0a7c6fe867c741589da parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/db60417f-cdd0-468c-805d-7ee3229735cc_calibrate_1123957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/db60417f-cdd0-468c-805d-7ee3229735cc_calibrate_1123957_17.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/5e87df75-122e-47b4-aa07-f01f935f8371_characterizeImage_1123957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/5e87df75-122e-47b4-aa07-f01f935f8371_characterizeImage_1123957_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 CG 16882983 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 217 parsl.dataflow.strategy DEBUG: Executor multi has 217 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 217 parsl.dataflow.strategy DEBUG: Executor multi has 217 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b70da54bcbe1719e2a00c3a3b528b58 with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 6a2a3c6f3f044035c4b540f64452df1e parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/324cf0a0-f1c6-40dd-aba9-328d48ef61d8_characterizeImage_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/324cf0a0-f1c6-40dd-aba9-328d48ef61d8_characterizeImage_1123955_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 0 failed parsl.dataflow.dflow INFO: Task 92 marked for retry parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash befb906dc3e3d9272113f719d8bae610 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6391f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 1 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b805c8d68401ae47a999552d269c681b with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash f77b5b0332781ed8ff845634e49deb65 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628098a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/121d81c3-af55-485b-b6a8-14499c5cc919_calibrate_1123955_46.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/121d81c3-af55-485b-b6a8-14499c5cc919_calibrate_1123955_46.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/9ef4c472-4628-4b75-94fe-f43bd7895c0f_characterizeImage_1123955_46.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/9ef4c472-4628-4b75-94fe-f43bd7895c0f_characterizeImage_1123955_46.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87bee4c0de33398420f729eca878d502 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 245d3d4e96494ed93dfaaaf2e581c763 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639db920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5cec7d02-66ee-4e29-9b88-3ddafbddbc3d_calibrate_1123957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5cec7d02-66ee-4e29-9b88-3ddafbddbc3d_calibrate_1123957_37.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6f764238-b62c-461c-b22c-4d89de1f9ecf_characterizeImage_1123957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6f764238-b62c-461c-b22c-4d89de1f9ecf_characterizeImage_1123957_37.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 463 try 1 failed parsl.dataflow.dflow ERROR: Task 463 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dc6f9dbb7f1a31cbfe08969ac84f85d4 with result from task 463 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 464 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 464 try 0 failed parsl.dataflow.dflow INFO: Task 464 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/278e1ca4-6113-458f-aa43-f6baec16d0be_calibrate_1124086_1.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/278e1ca4-6113-458f-aa43-f6baec16d0be_calibrate_1124086_1.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/a1a596d8-88f4-4228-b4df-29b6297a0790_characterizeImage_1124086_1.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 212953490d0acd361d42e62acbecfe7e with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 751b27b37bf31fa8f2da7a1577f23768 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/adab0a0d-d733-4a35-a83b-b4100573fb81_calibrate_1123955_51.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/adab0a0d-d733-4a35-a83b-b4100573fb81_calibrate_1123955_51.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/60638750-74ed-4a2f-8759-e70349dae14b_characterizeImage_1123955_51.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/60638750-74ed-4a2f-8759-e70349dae14b_characterizeImage_1123955_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 391 try 1 failed parsl.dataflow.dflow ERROR: Task 391 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 31b84a3b71164a0e0a55d387eb617bd3 with result from task 391 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 392 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 392 try 0 failed parsl.dataflow.dflow INFO: Task 392 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d15d2b14-5b33-4138-9064-2c96ed3bf9e1_calibrate_1124086_41.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d15d2b14-5b33-4138-9064-2c96ed3bf9e1_calibrate_1124086_41.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/75c507f4-9d41-412b-b4b5-d0712e0de4f9_characterizeImage_1124086_41.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 215 parsl.dataflow.strategy DEBUG: Executor multi has 215 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09f0b345cdc0d5812728b5c74eb0fb48 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 4e41a4d6227e2055309a050df2e35c8d parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2583a147-a0fb-4b14-8437-04c9ed030ce4_calibrate_1123953_19.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2583a147-a0fb-4b14-8437-04c9ed030ce4_calibrate_1123953_19.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/99e121d5-40dc-4659-a2d9-39825c48b34a_characterizeImage_1123953_19.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/99e121d5-40dc-4659-a2d9-39825c48b34a_characterizeImage_1123953_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 526 try 1 failed parsl.dataflow.dflow ERROR: Task 526 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e6c8f7a701148714415480c763f89556 with result from task 526 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 527 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 527 try 0 failed parsl.dataflow.dflow INFO: Task 527 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/fc3eaa65-5a30-4ba1-aa30-00712726b552_calibrate_1124086_11.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/fc3eaa65-5a30-4ba1-aa30-00712726b552_calibrate_1124086_11.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/dc9cb88d-9a18-4105-8f88-f0c71c2e5347_characterizeImage_1124086_11.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 448 try 1 failed parsl.dataflow.dflow ERROR: Task 448 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2c1c88107ce808a9d707329c2b4d7221 with result from task 448 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 449 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 449 try 0 failed parsl.dataflow.dflow INFO: Task 449 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/671b34e4-a3c0-4419-ae3c-4cb06ad5dc2c_calibrate_1124086_21.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/671b34e4-a3c0-4419-ae3c-4cb06ad5dc2c_calibrate_1124086_21.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/af246946-2358-485e-82a9-76636101a746_characterizeImage_1124086_21.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 603af55d92094939c23eac6d006feeb3 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash c07367c987b7c99e79cd2da1a634edce parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3879ce68-8b3f-4529-861d-79ed1c5cb47c_calibrate_1123957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3879ce68-8b3f-4529-861d-79ed1c5cb47c_calibrate_1123957_49.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e0005ede-a607-44b3-800b-31d0a0f29a70_characterizeImage_1123957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e0005ede-a607-44b3-800b-31d0a0f29a70_characterizeImage_1123957_49.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 541 try 1 failed parsl.dataflow.dflow ERROR: Task 541 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry eac9364a230113d22ceff5dd36acd291 with result from task 541 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 542 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 542 try 0 failed parsl.dataflow.dflow INFO: Task 542 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4016349e-5a46-4b94-ae45-eaa7122442da_calibrate_1124086_26.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4016349e-5a46-4b94-ae45-eaa7122442da_calibrate_1124086_26.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/826d7601-56a1-4bfb-ac5a-1cb8a83cab40_characterizeImage_1124086_26.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6109bfb4f8761b98c3ca8026e5682cf with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 14c73832bf5da3e87ccb83df678f7535 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627296c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6f1d08a6-681e-41c3-be06-e3fb5c09b6c3_calibrate_1123955_28.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6f1d08a6-681e-41c3-be06-e3fb5c09b6c3_calibrate_1123955_28.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d23eaed7-dc68-4e77-aa71-6ab543edea65_characterizeImage_1123955_28.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d23eaed7-dc68-4e77-aa71-6ab543edea65_characterizeImage_1123955_28.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b6281fdad5f4b39219f55c0350462fe with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 2a18cb4a6c3403276166c9ecd558d6c3 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/362731a4-ee2e-4251-b42d-35d701d4a852_calibrate_1123955_20.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/362731a4-ee2e-4251-b42d-35d701d4a852_calibrate_1123955_20.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c3484b53-588a-4972-b138-6d66e4e07437_characterizeImage_1123955_20.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/c3484b53-588a-4972-b138-6d66e4e07437_characterizeImage_1123955_20.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 192c71489aa7635d1852ebd5671fc35e with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash f1c39b107dfa6b7cc043d27d8e39fbdf parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/1a21f267-862b-48eb-bff3-3258f0bd100c_calibrate_1123955_15.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/1a21f267-862b-48eb-bff3-3258f0bd100c_calibrate_1123955_15.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/85ad0103-6fa2-4b2a-82be-596344d3386a_characterizeImage_1123955_15.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/85ad0103-6fa2-4b2a-82be-596344d3386a_characterizeImage_1123955_15.stderr 2024-03-07 00:43:27 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:43:28 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17e8ea096b6243368ca72ba73d85ae85 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 76c05d7a05936384ba7c763a199f840c parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a0f16f22-aa3c-4940-b100-45efca7d99f5_calibrate_1123957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a0f16f22-aa3c-4940-b100-45efca7d99f5_calibrate_1123957_26.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a317be8-6552-44b8-8422-994cb2e9349c_characterizeImage_1123957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6a317be8-6552-44b8-8422-994cb2e9349c_characterizeImage_1123957_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:29 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33dc19a5775839be1fd59fe2abec4431 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 01a2c1e5b42afff7eff658f2ed1a230b parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/567b1748-bfae-4d59-875a-a012d96ddeac_calibrate_1123955_44.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/567b1748-bfae-4d59-875a-a012d96ddeac_calibrate_1123955_44.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5a7625c-7036-49b7-a9cd-5812839634a6_characterizeImage_1123955_44.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e5a7625c-7036-49b7-a9cd-5812839634a6_characterizeImage_1123955_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 553 try 1 failed parsl.dataflow.dflow ERROR: Task 553 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d60753152da78404cb8d5f35dbe2f2a1 with result from task 553 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 554 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 554 try 0 failed parsl.dataflow.dflow INFO: Task 554 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/64441073-39f6-484e-81af-f2ee979445cc_calibrate_1124086_40.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/64441073-39f6-484e-81af-f2ee979445cc_calibrate_1124086_40.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e2b7e21a-04a5-4db9-a2b7-92351ff2ea02_characterizeImage_1124086_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:29 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 538 try 1 failed parsl.dataflow.dflow ERROR: Task 538 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d6df22c1985fdff01a2d8c71f3363c5a with result from task 538 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 539 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 539 try 0 failed parsl.dataflow.dflow INFO: Task 539 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/51fd0736-a5cb-4da4-b254-71dd78fa07d4_calibrate_1124086_55.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/51fd0736-a5cb-4da4-b254-71dd78fa07d4_calibrate_1124086_55.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c019e5b0-41d3-46d7-a3db-b0828d4ae4a5_characterizeImage_1124086_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:31 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:32 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce256f55a742c3297803833183a3324a with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash e076307994e9607b6464fb6c5e1fccc0 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/ba928b59-bcd4-4f2e-aa1a-a048db7192f4_calibrate_1123957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/ba928b59-bcd4-4f2e-aa1a-a048db7192f4_calibrate_1123957_30.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ac4f8612-829c-4958-8f4c-9417a421f66a_characterizeImage_1123957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ac4f8612-829c-4958-8f4c-9417a421f66a_characterizeImage_1123957_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:32 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 442 try 1 failed parsl.dataflow.dflow ERROR: Task 442 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1f93203f4e9f414f54589472d4f8fad8 with result from task 442 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:33 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 443 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 443 try 0 failed parsl.dataflow.dflow INFO: Task 443 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/8acebdf7-8e71-4cf4-b597-c2388b18fd68_calibrate_1124086_14.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/8acebdf7-8e71-4cf4-b597-c2388b18fd68_calibrate_1124086_14.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fead402-9c7b-4982-8b0a-9c71636ef0e5_characterizeImage_1124086_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:33 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:33 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc6e970bd064e1c3446d207b204bbd81 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash c39ff50af6e27312c7016120daab2fc6 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b8d52e7-85fb-488f-9bda-efa5c25577cb_calibrate_1123953_48.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b8d52e7-85fb-488f-9bda-efa5c25577cb_calibrate_1123953_48.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7f66162a-621b-4983-b7f2-7c87c3675807_characterizeImage_1123953_48.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/7f66162a-621b-4983-b7f2-7c87c3675807_characterizeImage_1123953_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:34 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:34 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 193 try 1 failed parsl.dataflow.dflow ERROR: Task 193 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c3d49b98236bbd759e12c588adbade8e with result from task 193 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 194 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 194 try 0 failed parsl.dataflow.dflow INFO: Task 194 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/20b7be06-dc0e-4e38-a6c6-e34591d664ee_calibrate_1124086_27.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/20b7be06-dc0e-4e38-a6c6-e34591d664ee_calibrate_1124086_27.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/7ebbf5d8-4bcc-472e-8824-fc21c7ef8cca_characterizeImage_1124086_27.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 523 try 1 failed parsl.dataflow.dflow ERROR: Task 523 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a9d1832a360c9a2c6233bb2591c7c2cb with result from task 523 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 524 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 524 try 0 failed parsl.dataflow.dflow INFO: Task 524 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/aec534ce-afae-4012-96f3-0ce662776a8d_calibrate_1124086_52.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/aec534ce-afae-4012-96f3-0ce662776a8d_calibrate_1124086_52.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c9a4ed22-3f90-4900-a574-7d7e3b740fa4_characterizeImage_1124086_52.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 394 try 1 failed parsl.dataflow.dflow ERROR: Task 394 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8f7ec507cde9853dd144343303d12d8c with result from task 394 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 395 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 395 try 0 failed parsl.dataflow.dflow INFO: Task 395 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/2533b636-28bb-4e36-8ab1-5f587b3b322f_calibrate_1124086_42.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/2533b636-28bb-4e36-8ab1-5f587b3b322f_calibrate_1124086_42.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e27065c1-1d7a-41f8-ae29-29806ba29191_characterizeImage_1124086_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:34 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:35 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dba6ce411e50df38730eb9676b98d70f with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash e3d2d0a029f17c9a3bd04da9f893272b parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62729d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9bb66c40-7a82-415b-b3ad-4c9fde20b50c_calibrate_1123957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9bb66c40-7a82-415b-b3ad-4c9fde20b50c_calibrate_1123957_60.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/3e3b12ea-f2df-476f-beee-2e237fb07aab_characterizeImage_1123957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/3e3b12ea-f2df-476f-beee-2e237fb07aab_characterizeImage_1123957_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:35 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:35 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 962d60cd93b253fee1f388eb851d2f5b with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 86625b193035f7f3db81c427f4f68105 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/db9b4989-088a-496b-92bd-afdf9f41fee3_calibrate_1123953_21.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/db9b4989-088a-496b-92bd-afdf9f41fee3_calibrate_1123953_21.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25382b56-5b0f-439f-9721-3ec5b044bede_characterizeImage_1123953_21.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/25382b56-5b0f-439f-9721-3ec5b044bede_characterizeImage_1123953_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:36 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1efec5af9dd411c496b315d62191b645 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 6fa7840da29e0b438bafce4c44a431f1 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/8ef5c751-47da-46d9-b081-5cc09f1659c6_calibrate_1123957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/8ef5c751-47da-46d9-b081-5cc09f1659c6_calibrate_1123957_6.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90524e26-ba6b-4ca0-b365-9dc42ec28545_characterizeImage_1123957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/90524e26-ba6b-4ca0-b365-9dc42ec28545_characterizeImage_1123957_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:36 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:36 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 496 try 1 failed parsl.dataflow.dflow ERROR: Task 496 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 04698d798d4bd343184502558f343f84 with result from task 496 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 497 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 497 try 0 failed parsl.dataflow.dflow INFO: Task 497 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/6726c722-2b81-4155-8a84-7ae4009eced6_calibrate_1124086_13.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/6726c722-2b81-4155-8a84-7ae4009eced6_calibrate_1124086_13.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/15683646-afbc-4751-9a5d-9a0e8757d480_characterizeImage_1124086_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:37 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:37 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:37 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:37 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c132fc371ea34e953ba2b64ee73c4ea1 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash ee7eb00d9b433d56bf9a32acd260d309 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/31e37cf7-9dfb-4b66-b654-af4785298a97_calibrate_1123957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/31e37cf7-9dfb-4b66-b654-af4785298a97_calibrate_1123957_9.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/86743632-7506-4d83-87b6-ca56508ecaca_characterizeImage_1123957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/86743632-7506-4d83-87b6-ca56508ecaca_characterizeImage_1123957_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:38 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:38 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e05f1f0e7011d444b14311e0baa8b73 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 681ea50919690615eea07d093f6a05e1 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5171c05c-354d-458d-8f27-37ae3c6fffd7_calibrate_1123957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5171c05c-354d-458d-8f27-37ae3c6fffd7_calibrate_1123957_46.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/7c0055a0-3556-4dc4-86df-c9ea3725dffc_characterizeImage_1123957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/7c0055a0-3556-4dc4-86df-c9ea3725dffc_characterizeImage_1123957_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:38 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:38 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:39 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:39 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:39 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 343 try 1 failed parsl.dataflow.dflow ERROR: Task 343 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ee6a5e8a80bc064569e6f60791c72a01 with result from task 343 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 344 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 344 try 0 failed parsl.dataflow.dflow INFO: Task 344 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:40 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4bc15dad-8f3f-4507-9492-905acfdddd0e_calibrate_1124086_23.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4bc15dad-8f3f-4507-9492-905acfdddd0e_calibrate_1124086_23.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0b6433dd-6d5a-44c1-ac44-fb2d218356f4_characterizeImage_1124086_23.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8f6ce2152f5945106dc31ac06385d57 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 0b2c354ca7063e049d2f36203ad2badb parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc01727e-f5dd-4698-aaa6-7c5911a48442_calibrate_1123953_50.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc01727e-f5dd-4698-aaa6-7c5911a48442_calibrate_1123953_50.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/666722eb-57ff-48ad-bb6b-e65b99d222ca_characterizeImage_1123953_50.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/666722eb-57ff-48ad-bb6b-e65b99d222ca_characterizeImage_1123953_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:40 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d064cea2d0ba266db62ec8663b27b261 with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 03f27093bcd0536b9f36f170ccfd0e02 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f0ef063f-ab12-494e-a453-be8a7565398e_calibrate_1123953_25.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f0ef063f-ab12-494e-a453-be8a7565398e_calibrate_1123953_25.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e5048c9b-61f1-4688-83aa-b49f93c3a403_characterizeImage_1123953_25.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/e5048c9b-61f1-4688-83aa-b49f93c3a403_characterizeImage_1123953_25.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5730bf1c137ef641a399d64c0016e00 with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 9b0898b48d7aa447b24f1f3f462e165c parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4acd0671-f5e8-4161-9ace-ba9c76ec89aa_calibrate_1123957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4acd0671-f5e8-4161-9ace-ba9c76ec89aa_calibrate_1123957_23.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ea8ca52c-7b9e-4b81-9cb8-a0a2436a2847_characterizeImage_1123957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/ea8ca52c-7b9e-4b81-9cb8-a0a2436a2847_characterizeImage_1123957_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:40 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:41 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:41 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 631 try 1 failed parsl.dataflow.dflow ERROR: Task 631 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2bf0cbd27c0283b36a0dad924269a652 with result from task 631 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 632 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 632 try 0 failed parsl.dataflow.dflow INFO: Task 632 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/f409ad61-3f1c-4fa0-9ccb-128711c804e8_calibrate_1123957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/f409ad61-3f1c-4fa0-9ccb-128711c804e8_calibrate_1123957_2.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/64ad537c-47e2-4c20-980d-c17762dced12_characterizeImage_1123957_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:41 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0bb9accdf55c033e400bbe35f9ee4f with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 3955c889519f577aa7e71080b4b382e6 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/50590d02-25f3-4c61-a5b4-ed47d08395a1_calibrate_1123957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/50590d02-25f3-4c61-a5b4-ed47d08395a1_calibrate_1123957_55.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1db1da34-bf7e-453e-9a0d-5a64837b29de_characterizeImage_1123957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/1db1da34-bf7e-453e-9a0d-5a64837b29de_characterizeImage_1123957_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:42 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:42 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:42 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:43 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:43 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:44 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:44 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:44 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0296ffc2b7f5ffdc2da21562474049a5 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash a9458640c6bdc4d1b6692fc5d445a9b9 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4fd29945-f42e-4820-8aef-41f383cece13_calibrate_1123957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4fd29945-f42e-4820-8aef-41f383cece13_calibrate_1123957_12.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/8fb0621b-fe9c-40e5-885d-de2f13bf2573_characterizeImage_1123957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/8fb0621b-fe9c-40e5-885d-de2f13bf2573_characterizeImage_1123957_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:45 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 508 try 1 failed parsl.dataflow.dflow ERROR: Task 508 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 454abd4bc0d448a57cdb310375233ce8 with result from task 508 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 509 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 509 try 0 failed parsl.dataflow.dflow INFO: Task 509 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/39348bb3-dc37-4b5d-b07e-1a6202346373_calibrate_1124086_29.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/39348bb3-dc37-4b5d-b07e-1a6202346373_calibrate_1124086_29.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5fffe7f8-6df5-42ba-8ddc-1d1cd28db02f_characterizeImage_1124086_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:45 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:45 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:46 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:46 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f013f847bf0b8125e7cb1d403b2259a3 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 301f802eaa605d22db2c0d1b4b1d6d88 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62776200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/48683a0c-07ad-445b-87c4-fa67dc3404c1_calibrate_1123955_34.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/48683a0c-07ad-445b-87c4-fa67dc3404c1_calibrate_1123955_34.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2d30c17d-55a7-46cd-adda-bbdfd74c55da_characterizeImage_1123955_34.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/2d30c17d-55a7-46cd-adda-bbdfd74c55da_characterizeImage_1123955_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:47 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:47 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:47 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:47 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:47 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:47 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1b417b96d2882b8c18fde2621871dcd with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash fd116fe7dec7d45609bc74a17bd3dde8 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/56df859a-4693-4d1c-8467-fc6f23d4ad0f_calibrate_1123955_24.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/56df859a-4693-4d1c-8467-fc6f23d4ad0f_calibrate_1123955_24.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5d6392f2-2dc2-42c1-8395-d98f8c69c778_characterizeImage_1123955_24.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5d6392f2-2dc2-42c1-8395-d98f8c69c778_characterizeImage_1123955_24.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 634 try 1 failed parsl.dataflow.dflow ERROR: Task 634 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 86f043d6c9387ab64c67a62550153e19 with result from task 634 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 635 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 635 try 0 failed parsl.dataflow.dflow INFO: Task 635 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/2a3d9c87-42fd-4d3b-aeea-89505f51c7c1_calibrate_1124086_2.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/2a3d9c87-42fd-4d3b-aeea-89505f51c7c1_calibrate_1124086_2.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/887fddc1-08eb-44ad-8cf8-d167024a2ab7_characterizeImage_1124086_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:48 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:48 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35db422d28192f7155de1425a9d73b32 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6f158d77-2485-4b50-adee-e0cc4c4f3450_calibrate_1123953_1.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6f158d77-2485-4b50-adee-e0cc4c4f3450_calibrate_1123953_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 CG 16882982 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:48 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5b4cc812a384c867a63dca4060509f2 with result from task 14 parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 CG 16882982 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:49 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6553cab2-02e3-4468-8042-1e3d03517bdf_calibrate_1123953_3.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6553cab2-02e3-4468-8042-1e3d03517bdf_calibrate_1123953_3.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3698b276ad4a6c04338b49bbfdf7d6ce with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/45234151-805b-48df-a5c7-98b4fc262198_calibrate_1123957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/45234151-805b-48df-a5c7-98b4fc262198_calibrate_1123957_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 CG 16882982 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:49 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 604 try 1 failed parsl.dataflow.dflow ERROR: Task 604 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a6034ae8e3b23f47abdc90cc9d93c832 with result from task 604 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 605 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 605 try 0 failed parsl.dataflow.dflow INFO: Task 605 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/f3dd159a-01e7-40ab-8daa-45de129df53b_calibrate_1124086_33.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/f3dd159a-01e7-40ab-8daa-45de129df53b_calibrate_1124086_33.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5f5babc0-40d0-4f9e-918f-e668d06563ab_characterizeImage_1124086_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 CG 16882982 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:49 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 CG 16882982 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:43:50 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882983 CG 16882982 CG 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 197 parsl.dataflow.strategy DEBUG: Executor multi has 197 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 197 parsl.dataflow.strategy DEBUG: Executor multi has 197 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ec436fb57766a003b66bd954cc0aadc with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 4786a9531cb792ba08b82b32b95e89a0 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fe53a576-5832-4a68-9dda-0997f78d5787_calibrate_1123955_8.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fe53a576-5832-4a68-9dda-0997f78d5787_calibrate_1123955_8.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f57f91fe-8616-454f-acdf-a11237775f68_characterizeImage_1123955_8.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/f57f91fe-8616-454f-acdf-a11237775f68_characterizeImage_1123955_8.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 652d169eb91c0347f0abcc445b794f6e with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash a1c3a70b9930dd2b20b786217d0f0c05 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3f43b769-5147-48b3-b7c1-8419707d0df7_calibrate_1123955_38.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3f43b769-5147-48b3-b7c1-8419707d0df7_calibrate_1123955_38.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35d847b0-4f4e-439c-b57b-171f7814a66c_characterizeImage_1123955_38.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/35d847b0-4f4e-439c-b57b-171f7814a66c_characterizeImage_1123955_38.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94113a165b0dfccfe817a46461c0e16b with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2e315c1f-b53e-4814-ac93-b0f427aeb4c6_calibrate_1123955_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2e315c1f-b53e-4814-ac93-b0f427aeb4c6_calibrate_1123955_41.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 529 try 1 failed parsl.dataflow.dflow ERROR: Task 529 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f7af245d36596ef4c4f9435c61894e2c with result from task 529 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 530 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 530 try 0 failed parsl.dataflow.dflow INFO: Task 530 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4eb573a7-a2bd-4670-8389-8c03f1b8e8b0_calibrate_1124086_37.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4eb573a7-a2bd-4670-8389-8c03f1b8e8b0_calibrate_1124086_37.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/48975c6b-1fab-44bd-8681-b8aa26e37675_characterizeImage_1124086_37.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adbe12072b5d76404ccdae40d13c8a94 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/445a1497-fad6-4ef1-a62c-86199764214e_calibrate_1123955_10.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/445a1497-fad6-4ef1-a62c-86199764214e_calibrate_1123955_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 475 try 1 failed parsl.dataflow.dflow ERROR: Task 475 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0f90651eb258afdb3d48d4b209f202d2 with result from task 475 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 476 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 476 try 0 failed parsl.dataflow.dflow INFO: Task 476 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/3a7dc53c-d368-4ba7-b51e-42cd9cbafacc_calibrate_1124086_59.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/3a7dc53c-d368-4ba7-b51e-42cd9cbafacc_calibrate_1124086_59.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ac62922b-e185-4222-a68a-145fca32a280_characterizeImage_1124086_59.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 610 try 1 failed parsl.dataflow.dflow ERROR: Task 610 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bd20a3c757288cac0aab6495a16f2a4b with result from task 610 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 611 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 611 try 0 failed parsl.dataflow.dflow INFO: Task 611 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/1b71d7f4-fd05-4a73-a23c-9176830c4f24_calibrate_1124086_60.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/1b71d7f4-fd05-4a73-a23c-9176830c4f24_calibrate_1124086_60.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/0cfe120a-dfc0-48da-a79e-6cf424530930_characterizeImage_1124086_60.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60e569991278c2ee259b806a7cd023b4 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a2e7f4f4-c1f9-4c2e-95ea-0d5c1a0ec1ac_calibrate_1123955_14.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a2e7f4f4-c1f9-4c2e-95ea-0d5c1a0ec1ac_calibrate_1123955_14.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22f6f6734a1f854f7ad5b365b97a9420 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3faae82b-d0b8-49c0-9af6-73c7b71aa0f1_calibrate_1123955_23.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3faae82b-d0b8-49c0-9af6-73c7b71aa0f1_calibrate_1123955_23.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f89bd548be9c52c9af04d8069b5a7d1e with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7021b942-0e24-4fe4-83e8-e0b028b5cdac_calibrate_1123953_62.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7021b942-0e24-4fe4-83e8-e0b028b5cdac_calibrate_1123953_62.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bf952a1551f924d6dc7b2295e695e94 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4eb87902-4477-4e10-bcb6-f10ea8829337_calibrate_1123957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4eb87902-4477-4e10-bcb6-f10ea8829337_calibrate_1123957_52.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 188 parsl.dataflow.strategy DEBUG: Executor multi has 188 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4f9a535b31ddf576060ba4d8dad174e with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/40c11632-5f6f-4f23-9b74-6b4f0f64dee0_calibrate_1123955_5.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/40c11632-5f6f-4f23-9b74-6b4f0f64dee0_calibrate_1123955_5.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adcf0eed01884b39de975fd14e2d8cf8 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash d294b8a6c4e574f0702e43a63ed0ad2d parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3569c69d-9bda-4c80-91ef-57f3102a8e4e_calibrate_1123953_17.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3569c69d-9bda-4c80-91ef-57f3102a8e4e_calibrate_1123953_17.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2924da02-44de-40b1-8417-4a20bb556815_characterizeImage_1123953_17.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/2924da02-44de-40b1-8417-4a20bb556815_characterizeImage_1123953_17.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d07ae8851050377f00e2af272bc9c2e with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 9e29e4559b2d94513f8520e1039fd502 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bf45ff94-954d-4110-9942-efa871d4f08f_characterizeImage_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/bf45ff94-954d-4110-9942-efa871d4f08f_characterizeImage_1123957_61.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e24c4528b28ada02f85e1a8adb7de09 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b509c223-503d-45db-8ecb-b4ab098c66a4_calibrate_1123955_6.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b509c223-503d-45db-8ecb-b4ab098c66a4_calibrate_1123955_6.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 612137701453aba31e72ca55512a9e8a with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f8f11e-ec7c-4b2e-9b10-89302a0eb9e5_calibrate_1123955_32.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f8f11e-ec7c-4b2e-9b10-89302a0eb9e5_calibrate_1123955_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 685 try 1 failed parsl.dataflow.dflow ERROR: Task 685 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1eb5af4c7b82d754d1de710d683f7fc7 with result from task 685 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 686 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 686 try 0 failed parsl.dataflow.dflow INFO: Task 686 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/dbc666d2-75ac-4fe5-b6e9-492ae652922f_calibrate_1124086_17.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/dbc666d2-75ac-4fe5-b6e9-492ae652922f_calibrate_1124086_17.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/292633dc-0566-41de-b90c-844147364294_characterizeImage_1124086_17.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 780246211932e09fe586d21dcdc4f47a with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/09837c47-f4ba-4016-a570-3a7d77d5d52e_calibrate_1123957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/09837c47-f4ba-4016-a570-3a7d77d5d52e_calibrate_1123957_29.stderr 2024-03-07 00:44:00 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:44:00 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caf9b8746a6e5379d78b2aaac687db2c with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/63949d4a-0945-4b8f-9f12-4abacbf57c7c_calibrate_1123955_36.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/63949d4a-0945-4b8f-9f12-4abacbf57c7c_calibrate_1123955_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:00 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc6b94b2f9f4514773c6f5a48465cc5 with result from task 709 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:01 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 462984884b78e060d3ec6e2b0694f13c parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/983c83c5-b445-468b-a02f-e6356a26fd92_calibrate_1123953_39.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/983c83c5-b445-468b-a02f-e6356a26fd92_calibrate_1123953_39.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cdddb0d9-f32b-4312-b1e9-72fd2fb9e02a_characterizeImage_1123953_39.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cdddb0d9-f32b-4312-b1e9-72fd2fb9e02a_characterizeImage_1123953_39.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08e6f19561a85b5b6a68f608b6f6d653 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 52bd24b34278da0a0e4e1b7499f98c9a parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62774c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/c768863b-504e-4a10-8b6b-205f96dae55c_calibrate_1123953_60.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/c768863b-504e-4a10-8b6b-205f96dae55c_calibrate_1123953_60.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3ddc56b0-ac20-407a-89bf-c4d5649a9687_characterizeImage_1123953_60.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/3ddc56b0-ac20-407a-89bf-c4d5649a9687_characterizeImage_1123953_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:01 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 556 try 1 failed parsl.dataflow.dflow ERROR: Task 556 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4b0fb9474321991e8acbdb77e04dd5de with result from task 556 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 557 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 557 try 0 failed parsl.dataflow.dflow INFO: Task 557 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/a826b18a-933f-46c7-b59a-9a305dbcedb0_calibrate_1124086_56.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/a826b18a-933f-46c7-b59a-9a305dbcedb0_calibrate_1124086_56.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d0aaae5a-27bc-4acd-8e59-1385d2c348c3_characterizeImage_1124086_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:01 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 347034fa272c219a2867ef754dd8ce9d with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6c82a894-d6b7-4409-b809-f317aba237b0_calibrate_1123955_59.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6c82a894-d6b7-4409-b809-f317aba237b0_calibrate_1123955_59.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99b36eb79dee1a263585f92e38446d7c with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a25caccb-d81c-448a-9cb0-8714cd7d284f_calibrate_1123953_28.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a25caccb-d81c-448a-9cb0-8714cd7d284f_calibrate_1123953_28.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3e8d57d82f9f29347c1357640fc354b with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 90cf519d8af55f4b2ef304a0857a8c59 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/251f99ec-4151-4c08-af1c-0d9f673af503_calibrate_1123953_55.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/251f99ec-4151-4c08-af1c-0d9f673af503_calibrate_1123953_55.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bfb0fb72-e688-4652-8ec2-fbb281c6cfa9_characterizeImage_1123953_55.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bfb0fb72-e688-4652-8ec2-fbb281c6cfa9_characterizeImage_1123953_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:04 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2061d9d9fcb7184328b7ef0b4cc5e0de with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/f93a04d4-5cbb-482a-9f04-b085c55d3f02_calibrate_1123957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/f93a04d4-5cbb-482a-9f04-b085c55d3f02_calibrate_1123957_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:04 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b712fb2e2e2bbb3bb906a622cf7bca00 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash acf129a380bae19f20cdbb373b101172 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7c37c4f1-6fd8-4b33-90b4-1c953d22c6e2_calibrate_1123953_59.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7c37c4f1-6fd8-4b33-90b4-1c953d22c6e2_calibrate_1123953_59.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/de71df5a-1e7f-4a66-a842-7e93aed7e0b4_characterizeImage_1123953_59.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/de71df5a-1e7f-4a66-a842-7e93aed7e0b4_characterizeImage_1123953_59.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 085f090bd960466ec3a93b3c61b81f89 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/96b6e4b9-41be-494c-875d-e125b82c1475_calibrate_1123953_40.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/96b6e4b9-41be-494c-875d-e125b82c1475_calibrate_1123953_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 640 try 1 failed parsl.dataflow.dflow ERROR: Task 640 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bf7a06ed9eb0a5b3afd1cd25772b0434 with result from task 640 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 641 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 641 try 0 failed parsl.dataflow.dflow INFO: Task 641 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d3daf87f-d6d5-4f3f-83cf-14604b12cb38_calibrate_1124086_3.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d3daf87f-d6d5-4f3f-83cf-14604b12cb38_calibrate_1124086_3.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/78cd6ba8-0f1c-4cc4-a9f7-16599a46c4b2_characterizeImage_1124086_3.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f655eaa02eee626544996ea7dd4119a with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/deeec811-24dd-47d5-9e90-cc47ed67c0bc_calibrate_1123953_44.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/deeec811-24dd-47d5-9e90-cc47ed67c0bc_calibrate_1123953_44.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68dd5d258fcf24e4b3782d931a500c43 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6e2a25c3-37c8-4d5b-b01e-1353cac44fb6_calibrate_1123953_46.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6e2a25c3-37c8-4d5b-b01e-1353cac44fb6_calibrate_1123953_46.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 479540b13b25978c268b7d83c7a3ee4d with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash ac9ba16794d3c3065a5091bbb597c17a parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/09b8f5bc-816a-416d-a57b-0e422437b708_calibrate_1123955_58.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/09b8f5bc-816a-416d-a57b-0e422437b708_calibrate_1123955_58.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ebab0828-1e5c-4902-8a18-6bf52c160cf9_characterizeImage_1123955_58.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ebab0828-1e5c-4902-8a18-6bf52c160cf9_characterizeImage_1123955_58.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1101529b540d820d0d1eb1aafc134a1 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 024dd4cf44f90b87239f0c14c1eff417 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7d86ce83-282f-40e0-b6be-a80ac0f3ad56_calibrate_1123957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7d86ce83-282f-40e0-b6be-a80ac0f3ad56_calibrate_1123957_38.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0ef6ffbc-2306-446f-9cce-e4dabfdf3805_characterizeImage_1123957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/0ef6ffbc-2306-446f-9cce-e4dabfdf3805_characterizeImage_1123957_38.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca5263249ecb940a11f8e6b0faf7bc64 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0f5c6571-f910-465a-9399-5a0dc22d907d_calibrate_1123955_47.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0f5c6571-f910-465a-9399-5a0dc22d907d_calibrate_1123955_47.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c5e3bbface9d92724222ad0d3eaf0a7 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9cf098bf-025e-4942-8128-e479cc1dd36f_calibrate_1123957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9cf098bf-025e-4942-8128-e479cc1dd36f_calibrate_1123957_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:06 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75b210d20d72db90488b8495c9844184 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a773cb8-d36d-4691-bc6c-8b8087939595_calibrate_1123953_30.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a773cb8-d36d-4691-bc6c-8b8087939595_calibrate_1123953_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:06 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3d5beebae92e410569e705713e10c6b with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/864bd408-4625-4bfc-9a69-7b149463e7ec_calibrate_1123953_6.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/864bd408-4625-4bfc-9a69-7b149463e7ec_calibrate_1123953_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:07 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3335aff30a081ce9082223a2aa02c3f with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash d3e49d4b8360b79fe74089677db1db38 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3e98ecb6-9021-4ece-b900-2f7ddf05a90a_calibrate_1123953_56.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3e98ecb6-9021-4ece-b900-2f7ddf05a90a_calibrate_1123953_56.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ad5232f4-12ad-4361-8f93-1cced134879d_characterizeImage_1123953_56.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/ad5232f4-12ad-4361-8f93-1cced134879d_characterizeImage_1123953_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:07 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd5a631cd809e1fd58d625f361886fe2 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 54226bfdaf5c6819fb56b490286d3b81 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/292c0022-cfc0-4169-a35a-5074984f5635_calibrate_1123955_7.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/292c0022-cfc0-4169-a35a-5074984f5635_calibrate_1123955_7.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/0bdcfb24-312a-4b8d-ae6f-70e5f9ed210a_characterizeImage_1123955_7.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/0bdcfb24-312a-4b8d-ae6f-70e5f9ed210a_characterizeImage_1123955_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:08 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:08 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:08 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:09 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:09 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 691 try 1 failed parsl.dataflow.dflow ERROR: Task 691 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry be24cf2f69351ea2ea79d0cb66c19d43 with result from task 691 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 692 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 692 try 0 failed parsl.dataflow.dflow INFO: Task 692 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/f00fefd2-10f3-477b-836b-a6a76ee164c0_calibrate_1124086_61.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/f00fefd2-10f3-477b-836b-a6a76ee164c0_calibrate_1124086_61.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/589b21d2-20be-46b7-baa2-9687a98f3904_characterizeImage_1124086_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:10 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0367e1f834e61815d78aeb1d4ea1a9fd with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash b5d27715cfe1d3fa68803a4564e55366 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a12b20e1-2aa1-40af-acfc-b820674d02a7_calibrate_1123955_50.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a12b20e1-2aa1-40af-acfc-b820674d02a7_calibrate_1123955_50.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e7f574c-71e9-497b-a5d2-c8442d175e75_characterizeImage_1123955_50.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5e7f574c-71e9-497b-a5d2-c8442d175e75_characterizeImage_1123955_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:10 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:10 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 673 try 1 failed parsl.dataflow.dflow ERROR: Task 673 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 48b5a3340b4cbb113a64d14de752a40b with result from task 673 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 674 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 674 try 0 failed parsl.dataflow.dflow INFO: Task 674 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d4af05b4-978c-4209-9829-4c3bc8f15179_calibrate_1124086_35.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/d4af05b4-978c-4209-9829-4c3bc8f15179_calibrate_1124086_35.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/ad391d24-bc32-4b94-a416-98033107d08b_characterizeImage_1124086_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:11 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4dc0a5ba90cd61a90feedc658a9dd5 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d7909e57-4f6f-4072-8362-874e3c4138e1_calibrate_1123953_9.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d7909e57-4f6f-4072-8362-874e3c4138e1_calibrate_1123953_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:11 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff9ec2860bb518ba5119b8f15a002d63 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c476ec8c-cda0-4193-b4c6-52d686f75fc0_calibrate_1123957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c476ec8c-cda0-4193-b4c6-52d686f75fc0_calibrate_1123957_41.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10934ebb98cd53f6a3ad7ba96900083d with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash b4f77adb25e6d50a676998bdc573f297 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b1b477a9-02e0-4579-a7da-d1df5b9dfb44_calibrate_1123957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b1b477a9-02e0-4579-a7da-d1df5b9dfb44_calibrate_1123957_39.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f761bebd-a901-4b5f-b7d5-283fa8547b44_characterizeImage_1123957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/f761bebd-a901-4b5f-b7d5-283fa8547b44_characterizeImage_1123957_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:11 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:12 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:12 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0634b741da09cdc6717f20bf940a46ce with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5dec0fd6-19a5-40f2-a21c-97a554c53f27_calibrate_1123955_13.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5dec0fd6-19a5-40f2-a21c-97a554c53f27_calibrate_1123955_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:13 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fed91f899d13206c1e6e3461e52cd20c with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ea0f96f5-11eb-4921-b63a-4856e5843dca_calibrate_1123955_40.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ea0f96f5-11eb-4921-b63a-4856e5843dca_calibrate_1123955_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 715 try 1 failed parsl.dataflow.dflow ERROR: Task 715 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4b2c0b62908ef6c5e468082bb994e87b with result from task 715 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 716 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 716 try 0 failed parsl.dataflow.dflow INFO: Task 716 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/69008de1-a283-4d31-9676-68f740b12a7f_calibrate_1124086_31.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/69008de1-a283-4d31-9676-68f740b12a7f_calibrate_1124086_31.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/f8eb9e22-e9b8-43a4-bba6-2c799a8946e5_characterizeImage_1124086_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:13 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:13 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a4edb35c52e6bdd6ae2e0717e15e769 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/49c2cc46-d244-43bd-b734-02e09549a7c8_calibrate_1123957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/49c2cc46-d244-43bd-b734-02e09549a7c8_calibrate_1123957_33.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff25dc2fac7ab8994269ae662569b9ee with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash e6425ba8f181cb392511c8dd66111a2e parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/58167e5f-1615-4eb8-8f8d-9a616b83d15c_calibrate_1123953_47.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/58167e5f-1615-4eb8-8f8d-9a616b83d15c_calibrate_1123953_47.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b7ead10f-8e52-4d9a-b3af-2aeb76c9da11_characterizeImage_1123953_47.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b7ead10f-8e52-4d9a-b3af-2aeb76c9da11_characterizeImage_1123953_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:14 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:14 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 151 try 1 failed parsl.dataflow.dflow ERROR: Task 151 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8a79d7bf52e2cbb96762550b3eb969f7 with result from task 151 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 152 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 152 try 0 failed parsl.dataflow.dflow INFO: Task 152 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/fa809352-6505-44cd-88aa-3584f1c0ead8_calibrate_1124086_18.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/fa809352-6505-44cd-88aa-3584f1c0ead8_calibrate_1124086_18.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/d752a87c-0681-477e-bba5-3d757822fd42_characterizeImage_1124086_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:14 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:15 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:15 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07efc31d3f502bf60e4e6ce087c45efe with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bff37879-148b-464b-97aa-6d8ac8a890d6_calibrate_1123955_27.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bff37879-148b-464b-97aa-6d8ac8a890d6_calibrate_1123955_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:15 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0684c7866c0b42b9b818b9bb98824cf1 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 11eff73aaf22d79bae89138a0a130d34 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f6ddd5d8-2e69-4973-8846-fd5d9300ffee_calibrate_1123953_22.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f6ddd5d8-2e69-4973-8846-fd5d9300ffee_calibrate_1123953_22.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c8b673b5-7c10-4bb7-9bdd-67a6a04e85b0_characterizeImage_1123953_22.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/c8b673b5-7c10-4bb7-9bdd-67a6a04e85b0_characterizeImage_1123953_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:16 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:16 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:16 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:17 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:17 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:17 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5a521a388f9323e8a697c531976d2ca with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 5cf4f066afd1bbea4dbbc6ea20f7c29f parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a2f52dc6-7cbe-4450-8887-6473e1f86721_calibrate_1123953_49.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a2f52dc6-7cbe-4450-8887-6473e1f86721_calibrate_1123953_49.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cfd24f69-0cb2-4250-9828-430c0006dfe5_characterizeImage_1123953_49.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/cfd24f69-0cb2-4250-9828-430c0006dfe5_characterizeImage_1123953_49.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 346 try 1 failed parsl.dataflow.dflow ERROR: Task 346 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1d8308824eaf28aad553a8cef558f49b with result from task 346 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 347 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 347 try 0 failed parsl.dataflow.dflow INFO: Task 347 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:18 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/bcb54ce6-5ad3-4aa2-81d4-b8879207b547_calibrate_1124086_50.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/bcb54ce6-5ad3-4aa2-81d4-b8879207b547_calibrate_1124086_50.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/5a93530e-51cd-40b5-b4b7-2e3e969fb20f_characterizeImage_1124086_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 667 try 1 failed parsl.dataflow.dflow ERROR: Task 667 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3ae032be8cd92868527afb26acdf35ec with result from task 667 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 668 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 668 try 0 failed parsl.dataflow.dflow INFO: Task 668 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/e34e707a-0c8d-47b8-bc8c-af6c7e284e52_calibrate_1124086_45.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/e34e707a-0c8d-47b8-bc8c-af6c7e284e52_calibrate_1124086_45.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/93383a25-ad15-438a-84b5-466151e4c908_characterizeImage_1124086_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:18 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:18 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:19 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:19 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:19 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:20 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 361 try 1 failed parsl.dataflow.dflow ERROR: Task 361 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a2539746568cd4d0c7e943d5339c97eb with result from task 361 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 362 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 362 try 0 failed parsl.dataflow.dflow INFO: Task 362 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/3406a5da-91d9-41a6-b058-97c14e2d3eb5_calibrate_1124086_43.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/3406a5da-91d9-41a6-b058-97c14e2d3eb5_calibrate_1124086_43.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/441f249b-081b-48a3-a4d5-87fd13f4f061_characterizeImage_1124086_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:20 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ddaa8a567267b4f9ef341c04243f6b0 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/25c8ebc9-02ca-4af6-8b06-83c94cf504ff_calibrate_1123953_42.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/25c8ebc9-02ca-4af6-8b06-83c94cf504ff_calibrate_1123953_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:20 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf41b7d0e7d7081aed66514e3ecd854 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cb1705d5-ddf0-4514-99f0-0d172719f1da_calibrate_1123957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cb1705d5-ddf0-4514-99f0-0d172719f1da_calibrate_1123957_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:21 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:21 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 325 try 1 failed parsl.dataflow.dflow ERROR: Task 325 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 520d9b2064cccd9ca720a96a980ca4c8 with result from task 325 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 326 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 326 try 0 failed parsl.dataflow.dflow INFO: Task 326 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/c58d6fa8-ab6f-40f9-a5cc-a7177275bed5_calibrate_1124086_34.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/c58d6fa8-ab6f-40f9-a5cc-a7177275bed5_calibrate_1124086_34.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/e55b5c7c-f7fb-4ffd-bfbf-0cd83ddd200f_characterizeImage_1124086_34.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0d5fadb1d1d45f48e7743882fc07ac6 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/df04b87c-d5b0-4a4b-a023-0c59188ac4f3_calibrate_1123953_36.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/df04b87c-d5b0-4a4b-a023-0c59188ac4f3_calibrate_1123953_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:21 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:22 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:22 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b446d44ed1dd4dab0366f4aca8a4d745 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bf460135-27cb-4f2b-b672-149830f37e48_calibrate_1123953_13.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bf460135-27cb-4f2b-b672-149830f37e48_calibrate_1123953_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:22 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39d3b26405abc4ca7beb981296397818 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash dd97fd4f396baa52e83a800640b03c20 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62728d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9420b5cd-282b-4896-b803-3504ea94aca0_calibrate_1123957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9420b5cd-282b-4896-b803-3504ea94aca0_calibrate_1123957_21.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e3921330-5f26-4366-83aa-aecc1b9a2d03_characterizeImage_1123957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/e3921330-5f26-4366-83aa-aecc1b9a2d03_characterizeImage_1123957_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:23 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:23 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:23 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b299f921ad60de691d19de9c6e6cc80b with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2c01fd70-b692-4f87-92e4-4780e22f124a_calibrate_1123953_8.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2c01fd70-b692-4f87-92e4-4780e22f124a_calibrate_1123953_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 367 try 1 failed parsl.dataflow.dflow ERROR: Task 367 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c26aa2901edd56434605a1b53a2f724f with result from task 367 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 368 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 368 try 0 failed parsl.dataflow.dflow INFO: Task 368 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/c55f239b-ca7a-4eed-9e44-34f060dba29b_calibrate_1124086_15.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/c55f239b-ca7a-4eed-9e44-34f060dba29b_calibrate_1124086_15.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/34a77822-d3f2-4c9d-90aa-b24618641def_characterizeImage_1124086_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:24 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:24 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27c7d93c049e3f6b3ea76e1c240039d3 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash e1e6028d4bf48d3f42fc27fc76bbfce6 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/ab126c75-d6e6-4cc1-9b44-d1356c3e483b_calibrate_1123953_58.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/ab126c75-d6e6-4cc1-9b44-d1356c3e483b_calibrate_1123953_58.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/90fcaa99-027a-49f5-80f1-f92257eedf28_characterizeImage_1123953_58.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/90fcaa99-027a-49f5-80f1-f92257eedf28_characterizeImage_1123953_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:24 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:25 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 150 parsl.dataflow.strategy DEBUG: Executor multi has 150 active tasks, 60/2 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 150 parsl.dataflow.strategy DEBUG: Executor multi has 150 active tasks, 60/2 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 002a2115097818868b99c5986819b9cf with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c09fabe0-1f9e-4b33-8427-77d1cb3e69a3_calibrate_1123957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c09fabe0-1f9e-4b33-8427-77d1cb3e69a3_calibrate_1123957_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 655 try 1 failed parsl.dataflow.dflow ERROR: Task 655 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a8c23f316931b637a7912e16787a011f with result from task 655 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 656 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 656 try 0 failed parsl.dataflow.dflow INFO: Task 656 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4bc54bf9-c4d3-4f0f-94cc-e377f1eeee95_calibrate_1124086_24.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/4bc54bf9-c4d3-4f0f-94cc-e377f1eeee95_calibrate_1124086_24.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/77645a2e-807f-4a96-bb28-ff74cea80d9a_characterizeImage_1124086_24.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b620463452968fedcfc6f95f8ee88e0f with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 46ff588c47f9d5586168e5c3ba5d2085 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b627282c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/76876c0a-a7e0-48a4-a4fb-6b71f08021c5_calibrate_1123953_12.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/76876c0a-a7e0-48a4-a4fb-6b71f08021c5_calibrate_1123953_12.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bc04d8dd-d713-4671-93fd-2b64688d4a53_characterizeImage_1123953_12.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/bc04d8dd-d713-4671-93fd-2b64688d4a53_characterizeImage_1123953_12.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d94d28cb4f3a38f28530210b64b9030c with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/34eea4f7-5188-478d-8026-baebebbfb025_calibrate_1123957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/34eea4f7-5188-478d-8026-baebebbfb025_calibrate_1123957_3.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae5bfa09a3fdfda730dba7034c2f12de with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cfcc7c8d-ede0-4c4a-85f7-57e23d60901d_calibrate_1123957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/cfcc7c8d-ede0-4c4a-85f7-57e23d60901d_calibrate_1123957_43.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dc4f1fb7fdb458c0289fd9d4a316ac7 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/2c82422a-7a89-4f4f-bcf6-40931b789c58_calibrate_1123957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/2c82422a-7a89-4f4f-bcf6-40931b789c58_calibrate_1123957_54.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 145 parsl.dataflow.strategy DEBUG: Executor multi has 145 active tasks, 60/2 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 838118bd48b06d2597b0237c03f83db5 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d58872ce-9d5e-4beb-ab66-a91433fa9d12_calibrate_1123955_52.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d58872ce-9d5e-4beb-ab66-a91433fa9d12_calibrate_1123955_52.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 346fe0a28fabf318c4e598f6cfe58e23 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash bab6aa62f9596265d05a7b92ff006323 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ffbfe150-79e5-4cf4-ab92-721fb7d52410_calibrate_1123955_3.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ffbfe150-79e5-4cf4-ab92-721fb7d52410_calibrate_1123955_3.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/646e2e0b-88cc-4844-a1a6-5aadc352a40f_characterizeImage_1123955_3.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/646e2e0b-88cc-4844-a1a6-5aadc352a40f_characterizeImage_1123955_3.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e28559432127ee3e8dc8db9e3e3428d with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b18c4571-ae32-47ac-95e5-cc58a9e36f1a_calibrate_1123957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b18c4571-ae32-47ac-95e5-cc58a9e36f1a_calibrate_1123957_50.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c73243fc7de9187a81325a8161922be2 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d8dd1eae-f213-4765-96c5-e9c8f83cba43_calibrate_1123953_4.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/d8dd1eae-f213-4765-96c5-e9c8f83cba43_calibrate_1123953_4.stderr 2024-03-07 00:44:35 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:44:35 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f818e813d99a66fd0409b16681867934 with result from task 62 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:35 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0459b698-3887-4923-a003-ef993625a7d9_calibrate_1123955_11.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/0459b698-3887-4923-a003-ef993625a7d9_calibrate_1123955_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:36 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:36 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:37 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:37 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8f5df28e2489c118a1a8cec1370c77b with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/331daff7-f003-4e52-a7e8-a6eaadc97ee7_calibrate_1123957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/331daff7-f003-4e52-a7e8-a6eaadc97ee7_calibrate_1123957_15.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2939f0d1ba6dd7bbf5925a7f55bf6c2a with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/25f6b895-a656-48cb-b299-587c127707c2_calibrate_1123955_12.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/25f6b895-a656-48cb-b299-587c127707c2_calibrate_1123955_12.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02cdd02d74c3bd19df59a9bf7fccea03 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/88776509-1190-4108-8e43-0d80bff54982_calibrate_1123955_48.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/88776509-1190-4108-8e43-0d80bff54982_calibrate_1123955_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:39 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:39 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef6bd765d4f2c2293ebab15b725a692 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/6914ae6d-228e-41f8-aa14-98ee6090f90d_calibrate_1123957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/6914ae6d-228e-41f8-aa14-98ee6090f90d_calibrate_1123957_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:40 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:41 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:41 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:41 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:42 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:42 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dc0b6d9ca23e0fcc2fe7ced4321cb4b with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a85fe8a3-9d3f-468e-b62b-02d6d620798c_calibrate_1123957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a85fe8a3-9d3f-468e-b62b-02d6d620798c_calibrate_1123957_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:42 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:43 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:43 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:43 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:44 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:44 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:44 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa929eaf7c125c07f54ddbbc6eb7e1b9 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/42cbffa2-b430-410f-9283-309ae8e04b9f_calibrate_1123957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/42cbffa2-b430-410f-9283-309ae8e04b9f_calibrate_1123957_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:45 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:45 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12b04df649440141125304743af12c68 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3abc537f-205b-41b9-bb93-069ec7ee0459_calibrate_1123953_37.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3abc537f-205b-41b9-bb93-069ec7ee0459_calibrate_1123953_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:46 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:46 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:46 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:47 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2abcc8cd70f95f7e2f3d5f19da8381 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/16c15251-3a71-4ff1-bc20-0b4c81e4e375_calibrate_1123957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/16c15251-3a71-4ff1-bc20-0b4c81e4e375_calibrate_1123957_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:47 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62518c8975a173d1a57e5adf474c5737 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 48d57ed48ef2ec4570127c258d2817ac parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/76f99030-2b1e-438e-8cf1-8e2c9fd5084d_calibrate_1123955_49.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/76f99030-2b1e-438e-8cf1-8e2c9fd5084d_calibrate_1123955_49.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/cfc0c300-964f-45bd-adfe-aa0281bedd47_characterizeImage_1123955_49.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/cfc0c300-964f-45bd-adfe-aa0281bedd47_characterizeImage_1123955_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:47 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:48 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:48 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:48 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:49 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:49 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:49 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 535 try 0 failed parsl.dataflow.dflow INFO: Task 535 marked for retry parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 110da3a3ae3d9fc14dbb7e0f0a8b0ab6 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62898b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 1 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 571 try 0 failed parsl.dataflow.dflow INFO: Task 571 marked for retry parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 1ed1e24022251cfbcf51f147c187a4cd parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 1 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 562 try 0 failed parsl.dataflow.dflow INFO: Task 562 marked for retry parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 1857f6e1e5a64c266aa2be81cf1c40eb parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 1 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 565 try 0 failed parsl.dataflow.dflow INFO: Task 565 marked for retry parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash f798b034d9f2b31e11ae711acfa73dcd parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 1 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 592 try 0 failed parsl.dataflow.dflow INFO: Task 592 marked for retry parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 6474b7c372920189ee79de61d33b7648 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 1 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 586 try 0 failed parsl.dataflow.dflow INFO: Task 586 marked for retry parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 54b4fdbd1fe314ca6b36a95a66708743 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 1 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 433 try 0 failed parsl.dataflow.dflow INFO: Task 433 marked for retry parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 1c20e1eead23c0304b3b5a9fdecf68b2 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6280a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 1 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 577 try 0 failed parsl.dataflow.dflow INFO: Task 577 marked for retry parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 46005d271901f66057883f74f687121f parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 1 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:49 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:50 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:50 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 496dd38d4b50b8b80b95f5d9b2dac966 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/40df4906-d547-4f56-b0ac-c7e0cf5f0309_calibrate_1123957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/40df4906-d547-4f56-b0ac-c7e0cf5f0309_calibrate_1123957_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:51 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:51 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bc608979ada3045d63fe30df481041d with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/98fb946a-8981-4af8-b665-69199fe5abd6_calibrate_1123957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/98fb946a-8981-4af8-b665-69199fe5abd6_calibrate_1123957_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:51 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b843147e6d565e7611a6cb5d4899f97 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/4367cb71-7280-465d-9cd9-927e0040c001_calibrate_1123955_53.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/4367cb71-7280-465d-9cd9-927e0040c001_calibrate_1123955_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:51 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:52 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20d23be6a486b74a111e51c8d54409a9 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a172caa0-21e4-4704-8748-f2579dd0c599_calibrate_1123955_17.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a172caa0-21e4-4704-8748-f2579dd0c599_calibrate_1123955_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:52 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 247 try 0 failed parsl.dataflow.dflow INFO: Task 247 marked for retry parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 23c069ba06489649241b3161ff4213b1 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 1 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 337 try 0 failed parsl.dataflow.dflow INFO: Task 337 marked for retry parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash b1f3810d6e2c064aa39e39a410e7fe46 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639afc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 1 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 379 try 0 failed parsl.dataflow.dflow INFO: Task 379 marked for retry parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 0d2b40c17f5ab26685b9ad1441a8518b parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 1 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 622 try 0 failed parsl.dataflow.dflow INFO: Task 622 marked for retry parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash c3791ad784fa82a0f48917c19df80976 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 1 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 736 try 0 failed parsl.dataflow.dflow INFO: Task 736 marked for retry parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 2380e60c745ae607acf1ba38f5be5cdb parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 1 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 272 try 0 failed parsl.dataflow.dflow INFO: Task 272 marked for retry parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 7e60938d64a53629102c397040f026cb parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 1 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 125 try 0 failed parsl.dataflow.dflow INFO: Task 125 marked for retry parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash c09dca698effbf89db7ba0b4d78d7035 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 1 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 332 try 0 failed parsl.dataflow.dflow INFO: Task 332 marked for retry parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash cb52037a0a4744cd8457213b0a84d2e8 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639af420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 1 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:52 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:53 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:53 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d58dd3a599c1456c2ade4ea45c2f0b23 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash cff5844fcbc3c1e3f8f4aa2308050baf parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639d8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/df6c2d8d-168f-455e-83e4-97a2625505c0_calibrate_1123955_54.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/df6c2d8d-168f-455e-83e4-97a2625505c0_calibrate_1123955_54.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1caf4bd8-ca06-4c3b-bcc6-76f7134b98ca_characterizeImage_1123955_54.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/1caf4bd8-ca06-4c3b-bcc6-76f7134b98ca_characterizeImage_1123955_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:53 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 1 failed parsl.dataflow.dflow ERROR: Task 92 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry befb906dc3e3d9272113f719d8bae610 with result from task 92 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0922dcad-98cc-4c38-9fba-1b047d42e047_calibrate_1123953_31.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc307e51bb1a4c94cf2107f8acaa59d4 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/374739bb-fbc4-4cd3-a2a7-52f3fd5d7ce9_calibrate_1123955_43.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/374739bb-fbc4-4cd3-a2a7-52f3fd5d7ce9_calibrate_1123955_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:54 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e730e01d8b1b7017a14f74a53a37f43f with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5e0b7c4e-38c4-4358-994e-c8253e7acca1_calibrate_1123955_60.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5e0b7c4e-38c4-4358-994e-c8253e7acca1_calibrate_1123955_60.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b05b9abc31c4f66349c4f6420f45683 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d9967d76-e397-468c-b766-214e160c8d3b_calibrate_1123955_45.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/d9967d76-e397-468c-b766-214e160c8d3b_calibrate_1123955_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:54 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:54 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:55 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:55 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aeea6089ac98023aeb4c6309834e2c6 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/95084901-507b-49b7-b803-708a338b33c8_calibrate_1123953_33.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/95084901-507b-49b7-b803-708a338b33c8_calibrate_1123953_33.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 689 try 0 failed parsl.dataflow.dflow INFO: Task 689 marked for retry parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 6a2a3c6f3f044035c4b540f64452df1e parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 1 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:55 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:56 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cd6908cc5d2dec3d94d1717328d9766 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f73fa3-63a3-4194-91a1-c26d33838dd9_calibrate_1123955_37.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/52f73fa3-63a3-4194-91a1-c26d33838dd9_calibrate_1123955_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:56 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:56 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:57 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:57 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:57 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:44:57 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb2faff98240ad1860b69f4417e0f385 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e01252c5-434b-43d7-88d4-6f883730cddd_calibrate_1123953_24.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e01252c5-434b-43d7-88d4-6f883730cddd_calibrate_1123953_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 122 parsl.dataflow.strategy DEBUG: Executor multi has 122 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 122 parsl.dataflow.strategy DEBUG: Executor multi has 122 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2491803816ae949c44c7659679e466e with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0b83a737-5455-4bbb-be00-d120d5a4bc3f_calibrate_1123957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0b83a737-5455-4bbb-be00-d120d5a4bc3f_calibrate_1123957_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 266 try 0 failed parsl.dataflow.dflow INFO: Task 266 marked for retry parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 9bcc33a4f38ca75545a81815580759b9 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6398af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 1 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba3de698432dbb4d5ae5edbd4d17926 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/708272f4-3759-4997-a4b1-e36e1446410a_calibrate_1123955_25.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/708272f4-3759-4997-a4b1-e36e1446410a_calibrate_1123955_25.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f8c6f6d1562416fa7d3150f82b610bd with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/9efcf8ee-565a-4790-9d6a-28c2fa160061_calibrate_1123953_57.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/9efcf8ee-565a-4790-9d6a-28c2fa160061_calibrate_1123953_57.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0080da7e97d9c1d2a43ecd2e4ce97377 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b3e3ff9-a4be-4d89-a02f-7ebfc583a8ee_calibrate_1123953_53.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b3e3ff9-a4be-4d89-a02f-7ebfc583a8ee_calibrate_1123953_53.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 515b7f71b1657df8daca67be275de806 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/43b1924c-2846-486d-b484-5213f89b0a21_calibrate_1123957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/43b1924c-2846-486d-b484-5213f89b0a21_calibrate_1123957_45.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98ace61dd03d6a2e3878216f3a8358a8 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b0fbdf4f-e61b-4619-8367-808aa0c6706c_calibrate_1123957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b0fbdf4f-e61b-4619-8367-808aa0c6706c_calibrate_1123957_53.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ec5f854783a7b82debbeb0ba31e3f03 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/61390964-40c1-4469-80ec-c49e39b716f9_calibrate_1123957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/61390964-40c1-4469-80ec-c49e39b716f9_calibrate_1123957_1.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a652c93b9f7cebe6812bbfe3d14d4492 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7f488037-0ddd-4048-9ae8-4896b5ccc5c4_calibrate_1123955_57.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7f488037-0ddd-4048-9ae8-4896b5ccc5c4_calibrate_1123955_57.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 114 parsl.dataflow.strategy DEBUG: Executor multi has 114 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 530a20dbf36bf2efd3976d9842f478e4 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/053082a8-b406-4769-91c0-8d4aa781a32a_calibrate_1123955_18.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/053082a8-b406-4769-91c0-8d4aa781a32a_calibrate_1123955_18.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9c3f8af11c0e52018d7ad1c33b23eb8 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/33f890d2-ecd8-4be5-af7b-0d6626138010_calibrate_1123957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/33f890d2-ecd8-4be5-af7b-0d6626138010_calibrate_1123957_48.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6e731e491527e9cb9742c412e9c8bf9 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c77a9054-4b2e-4584-8c39-fd3958337811_calibrate_1123957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c77a9054-4b2e-4584-8c39-fd3958337811_calibrate_1123957_25.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6069a93b575e2134092d77ad224e7c6e with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5f91d19b-2c7a-4baf-b089-d54b238dcf5e_calibrate_1123955_62.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/5f91d19b-2c7a-4baf-b089-d54b238dcf5e_calibrate_1123955_62.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e85594ad063e8b5b4676018898a066 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f1ae02ce-080b-4167-853b-ac928223ef69_calibrate_1123953_54.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f1ae02ce-080b-4167-853b-ac928223ef69_calibrate_1123953_54.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6053c942044850a96282a7047e48a284 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bfff94f0-be20-447d-9021-c3c3c4debe65_calibrate_1123953_34.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bfff94f0-be20-447d-9021-c3c3c4debe65_calibrate_1123953_34.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fee5efa37c1286fd80aa46b372aa412c with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a4a41427-24a4-4718-ae72-9eeaabba067e_calibrate_1123955_9.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a4a41427-24a4-4718-ae72-9eeaabba067e_calibrate_1123955_9.stderr 2024-03-07 00:45:08 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:45:08 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8472ea7e99e054b486ab560b47ac5d24 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/327b1949-6f9a-49e6-932d-15c7f90f3331_calibrate_1123957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/327b1949-6f9a-49e6-932d-15c7f90f3331_calibrate_1123957_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:08 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d9c2e2365f5f126ac92b1bb10f522ad with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/37f2ce68-cf0d-4ced-b092-4ccb0040739c_calibrate_1123957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/37f2ce68-cf0d-4ced-b092-4ccb0040739c_calibrate_1123957_58.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19db1fc17d0a805d4e3275d28a5cfeea with result from task 482 parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:08 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3313c30a-e3fb-43bb-aa3a-3851757022fa_calibrate_1123953_16.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3313c30a-e3fb-43bb-aa3a-3851757022fa_calibrate_1123953_16.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7c89ca0226815d96c9a84960e0ea621 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b72b3486-bed6-4ae9-b952-db21dbb26e74_calibrate_1123955_39.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b72b3486-bed6-4ae9-b952-db21dbb26e74_calibrate_1123955_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:09 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:09 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:09 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9f3422d4ac8558de81133455885c4e8 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/fe40d186-f8c4-4a4e-b342-6aa559f39243_calibrate_1123953_35.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/fe40d186-f8c4-4a4e-b342-6aa559f39243_calibrate_1123953_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:10 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:10 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d054d10815c5bfd8006cf201038b2a3b with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e016df84-d5fa-45a1-975e-2d7b192a5f1f_calibrate_1123953_29.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e016df84-d5fa-45a1-975e-2d7b192a5f1f_calibrate_1123953_29.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7189a504755be783506fb510cc21f8d0 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b66b485f-e5ff-4c00-bce6-776669f54473_calibrate_1123955_16.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b66b485f-e5ff-4c00-bce6-776669f54473_calibrate_1123955_16.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc705b6a746c32471c8f363ab1bb8b26 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc6536b2-d843-42e1-8643-5ef30294e03d_calibrate_1123953_52.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc6536b2-d843-42e1-8643-5ef30294e03d_calibrate_1123953_52.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a18117defb7b018436ab0d6d0d3e0724 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/871e00c6-1e8f-4390-9c86-6935443824f4_calibrate_1123953_32.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/871e00c6-1e8f-4390-9c86-6935443824f4_calibrate_1123953_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:12 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:13 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8988013613219bd930132cf987bf8c68 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e0d2d8e8-e8bf-4d8b-b1e4-9ec6baceba68_calibrate_1123953_14.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/e0d2d8e8-e8bf-4d8b-b1e4-9ec6baceba68_calibrate_1123953_14.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcd8eb22aa5d63933c9eb15045819510 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c7573631-353b-406e-b2fa-4d430edade10_calibrate_1123957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/c7573631-353b-406e-b2fa-4d430edade10_calibrate_1123957_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:14 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:14 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 635feb60cc29de8a31386bf9308382b7 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/45493781-592a-49a4-9991-2120f57aef8d_calibrate_1123953_11.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/45493781-592a-49a4-9991-2120f57aef8d_calibrate_1123953_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:15 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:15 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a947ab5ec9c0d4280583a19a7751c097 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/54682175-cfdb-43a8-ab9c-adb3a328b0ed_calibrate_1123953_26.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/54682175-cfdb-43a8-ab9c-adb3a328b0ed_calibrate_1123953_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:15 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:16 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49797cfd98ae0bced97fb9e47b72b2c8 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/41f76e45-6ccb-44d0-893b-afef88025748_calibrate_1123955_29.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/41f76e45-6ccb-44d0-893b-afef88025748_calibrate_1123955_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:16 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:16 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:17 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f353dc895b460be666d92d69025d249 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/38ea1ffd-8081-4408-8057-8bde56d9c1b5_calibrate_1123957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/38ea1ffd-8081-4408-8057-8bde56d9c1b5_calibrate_1123957_51.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8e26ad24ed407b8439d4d2736a81bcb with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/09d507d9-a78d-414c-8124-5926cf79f1cb_calibrate_1123953_10.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/09d507d9-a78d-414c-8124-5926cf79f1cb_calibrate_1123953_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:17 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd74386b5a469c21343405e10900c8ec with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/73c475c4-75f3-4e97-98dd-7d364ecfe23b_calibrate_1123957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/73c475c4-75f3-4e97-98dd-7d364ecfe23b_calibrate_1123957_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:17 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:18 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0344ab7a6b29e7958ad2aa196db309cb with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0fa0cd5d-9289-40fe-a437-72e277534010_calibrate_1123953_45.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/0fa0cd5d-9289-40fe-a437-72e277534010_calibrate_1123953_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:18 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01bc92a6916d3997cef87d078425db88 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/291ac453-1482-48f6-95ea-2d993bce613d_calibrate_1123957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/291ac453-1482-48f6-95ea-2d993bce613d_calibrate_1123957_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:18 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b588ba0c2d20b23b2901d4895f1ae3a with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3a358e1b-dbbf-4426-80cb-062b0fbacc17_calibrate_1123955_22.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3a358e1b-dbbf-4426-80cb-062b0fbacc17_calibrate_1123955_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:19 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:19 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79b54366a22fcbfbda1d9bab16386963 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3e97db99-93e9-4b38-bb07-3abf4cc54e44_calibrate_1123957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3e97db99-93e9-4b38-bb07-3abf4cc54e44_calibrate_1123957_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:19 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:20 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:20 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:20 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 694 try 1 failed parsl.dataflow.dflow ERROR: Task 694 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 54c0a4e6f31cd52c053b2731c5493b68 with result from task 694 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 695 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 695 try 0 failed parsl.dataflow.dflow INFO: Task 695 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/3d0d922f-6b20-4aae-8825-e1e6d6d5d9f8_calibrate_1124086_22.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1124086/3d0d922f-6b20-4aae-8825-e1e6d6d5d9f8_calibrate_1124086_22.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1124086/c2aead31-c478-4b8a-afa5-92a989c7418c_characterizeImage_1124086_22.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 157a7e1bb8446b1babd71efe2e15d9ac with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/74387827-6623-4e77-a828-a290e7f4f998_calibrate_1123957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/74387827-6623-4e77-a828-a290e7f4f998_calibrate_1123957_34.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bf6d1141d6adb6a88b95d953aeb0a01 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fb7cc54f-be81-4200-82fe-e76211a3e9bd_calibrate_1123955_35.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fb7cc54f-be81-4200-82fe-e76211a3e9bd_calibrate_1123955_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:21 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:21 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:21 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 418ffff0d25a5edb91bacf754d54d918 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 39f461a91777e4a9e73eebfcc835ee5b parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6272bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/181dbba6-c2c0-4018-8cd5-698e7d857a49_calibrate_1123957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/181dbba6-c2c0-4018-8cd5-698e7d857a49_calibrate_1123957_57.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4abfa646-3475-461c-bbb9-a2ca9036eb33_characterizeImage_1123957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/4abfa646-3475-461c-bbb9-a2ca9036eb33_characterizeImage_1123957_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:22 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dedd728d5442c728ff8c8d966c3da3d with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5ec6c6f7-aad3-4da8-95e2-01ba2dbb9c60_calibrate_1123953_7.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5ec6c6f7-aad3-4da8-95e2-01ba2dbb9c60_calibrate_1123953_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:22 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:22 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:22 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14e10ac81d9b0e5d822cf2d1b355a62a with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6475e2ef-ad1c-402b-8b8c-7a0d7bedea45_calibrate_1123953_41.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6475e2ef-ad1c-402b-8b8c-7a0d7bedea45_calibrate_1123953_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:23 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dea23dea5e470be92b04e427840037f with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7017832d-1ff5-4f42-8f4d-e5e0c460bc60_calibrate_1123957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7017832d-1ff5-4f42-8f4d-e5e0c460bc60_calibrate_1123957_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:23 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:23 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:24 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf40222348efc767323285e14bc484cb with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4f87ca71-d52b-4b32-bd99-4d6fa22442c5_calibrate_1123953_27.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4f87ca71-d52b-4b32-bd99-4d6fa22442c5_calibrate_1123953_27.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 674e9668989ec2df423d68686ffa3a02 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/47a4a806-c7d8-49ca-bf9d-ed45a852a749_calibrate_1123953_51.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/47a4a806-c7d8-49ca-bf9d-ed45a852a749_calibrate_1123953_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:24 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:24 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dcfaed311fb6fbaf5a90cc560557976 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bdf69a5a-f7c3-4479-bdab-67222f2cb1aa_calibrate_1123955_21.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bdf69a5a-f7c3-4479-bdab-67222f2cb1aa_calibrate_1123955_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:25 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:25 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:26 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:26 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:26 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:27 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:27 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:27 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cedfdfe82434902774ba45c4f6f192e5 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3bbc640a-6189-4590-a02c-d1eca55083df_calibrate_1123955_1.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3bbc640a-6189-4590-a02c-d1eca55083df_calibrate_1123955_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:28 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:28 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cb3041605c2dbde93c7084a98fe39f3 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/be839559-212a-4b61-b3b0-0ddec2667a09_calibrate_1123957_8.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/be839559-212a-4b61-b3b0-0ddec2667a09_calibrate_1123957_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:28 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 245d3d4e96494ed93dfaaaf2e581c763 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5cec7d02-66ee-4e29-9b88-3ddafbddbc3d_calibrate_1123957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5cec7d02-66ee-4e29-9b88-3ddafbddbc3d_calibrate_1123957_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:29 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0695596058db808e60d9099d0dbaa292 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/84bce3ea-bddb-44bb-807e-3fc408487232_calibrate_1123957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/84bce3ea-bddb-44bb-807e-3fc408487232_calibrate_1123957_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:29 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:30 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 767c7177f762e448d293736ff9c18928 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5f3a6c1a-85b6-4bea-b714-0a3135a76734_calibrate_1123953_38.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/5f3a6c1a-85b6-4bea-b714-0a3135a76734_calibrate_1123953_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:30 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:30 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:31 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3c377554d80220dea477219e0452c08 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7011f473-a877-4a0c-a1c3-b4364c304be9_calibrate_1123955_30.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/7011f473-a877-4a0c-a1c3-b4364c304be9_calibrate_1123955_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:31 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 71 parsl.dataflow.strategy DEBUG: Executor multi has 71 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 71 parsl.dataflow.strategy DEBUG: Executor multi has 71 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4d2d0261a37b94b239dc235b86e5af4 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3d31e96e-9914-4fd3-bd9c-a4c7d19a77ac_calibrate_1123957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3d31e96e-9914-4fd3-bd9c-a4c7d19a77ac_calibrate_1123957_35.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 751b27b37bf31fa8f2da7a1577f23768 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/adab0a0d-d733-4a35-a83b-b4100573fb81_calibrate_1123955_51.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/adab0a0d-d733-4a35-a83b-b4100573fb81_calibrate_1123955_51.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e41a4d6227e2055309a050df2e35c8d with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2583a147-a0fb-4b14-8437-04c9ed030ce4_calibrate_1123953_19.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/2583a147-a0fb-4b14-8437-04c9ed030ce4_calibrate_1123953_19.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 515696ec91d8367d5113e93b41b905b8 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/fdc8b17d-b77d-4fc9-b84d-a184b3ada5f0_calibrate_1123957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/fdc8b17d-b77d-4fc9-b84d-a184b3ada5f0_calibrate_1123957_4.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c07367c987b7c99e79cd2da1a634edce with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3879ce68-8b3f-4529-861d-79ed1c5cb47c_calibrate_1123957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3879ce68-8b3f-4529-861d-79ed1c5cb47c_calibrate_1123957_49.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 66 parsl.dataflow.strategy DEBUG: Executor multi has 66 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53500d9b41450f6303fd9cd15b0ff6b4 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6d840c1e-0e3f-4abc-82cc-a3a0482f9fef_calibrate_1123953_15.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/6d840c1e-0e3f-4abc-82cc-a3a0482f9fef_calibrate_1123953_15.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad886ae1ef6ef69a2704742d185ddf87 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 897c6ed585c1c2391314d14e8eda8835 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6284e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/69f0324c-e28b-4b60-9629-c72f803ac8a1_calibrate_1123953_20.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/69f0324c-e28b-4b60-9629-c72f803ac8a1_calibrate_1123953_20.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1450c694-0d8b-49ec-895d-199e43554d2d_characterizeImage_1123953_20.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/1450c694-0d8b-49ec-895d-199e43554d2d_characterizeImage_1123953_20.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a18cb4a6c3403276166c9ecd558d6c3 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/362731a4-ee2e-4251-b42d-35d701d4a852_calibrate_1123955_20.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/362731a4-ee2e-4251-b42d-35d701d4a852_calibrate_1123955_20.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e076307994e9607b6464fb6c5e1fccc0 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/ba928b59-bcd4-4f2e-aa1a-a048db7192f4_calibrate_1123957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/ba928b59-bcd4-4f2e-aa1a-a048db7192f4_calibrate_1123957_30.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1c39b107dfa6b7cc043d27d8e39fbdf with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/1a21f267-862b-48eb-bff3-3258f0bd100c_calibrate_1123955_15.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/1a21f267-862b-48eb-bff3-3258f0bd100c_calibrate_1123955_15.stderr 2024-03-07 00:45:41 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:45:41 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:41 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:41 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:41 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3d2d0a029f17c9a3bd04da9f893272b with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9bb66c40-7a82-415b-b3ad-4c9fde20b50c_calibrate_1123957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9bb66c40-7a82-415b-b3ad-4c9fde20b50c_calibrate_1123957_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:42 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 272 try 1 failed parsl.dataflow.dflow ERROR: Task 272 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7e60938d64a53629102c397040f026cb with result from task 272 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5020381e-eb37-438b-b928-4a2e4160739d_calibrate_1123957_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:43 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:44 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:44 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86625b193035f7f3db81c427f4f68105 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/db9b4989-088a-496b-92bd-afdf9f41fee3_calibrate_1123953_21.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/db9b4989-088a-496b-92bd-afdf9f41fee3_calibrate_1123953_21.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01a2c1e5b42afff7eff658f2ed1a230b with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/567b1748-bfae-4d59-875a-a012d96ddeac_calibrate_1123955_44.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/567b1748-bfae-4d59-875a-a012d96ddeac_calibrate_1123955_44.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14c73832bf5da3e87ccb83df678f7535 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6f1d08a6-681e-41c3-be06-e3fb5c09b6c3_calibrate_1123955_28.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6f1d08a6-681e-41c3-be06-e3fb5c09b6c3_calibrate_1123955_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 689 try 1 failed parsl.dataflow.dflow ERROR: Task 689 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6a2a3c6f3f044035c4b540f64452df1e with result from task 689 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/b84ad35a-4fd6-4736-80ca-1b67fdc48642_calibrate_1123955_31.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c39ff50af6e27312c7016120daab2fc6 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b8d52e7-85fb-488f-9bda-efa5c25577cb_calibrate_1123953_48.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/8b8d52e7-85fb-488f-9bda-efa5c25577cb_calibrate_1123953_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:46 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:46 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:46 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:46 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3955c889519f577aa7e71080b4b382e6 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/50590d02-25f3-4c61-a5b4-ed47d08395a1_calibrate_1123957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/50590d02-25f3-4c61-a5b4-ed47d08395a1_calibrate_1123957_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:47 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:47 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:48 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:49 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:50 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:51 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fa7840da29e0b438bafce4c44a431f1 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/8ef5c751-47da-46d9-b081-5cc09f1659c6_calibrate_1123957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/8ef5c751-47da-46d9-b081-5cc09f1659c6_calibrate_1123957_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:51 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:51 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:51 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:51 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:51 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:51 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:45:52 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 PD 16882983 PD 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 53 parsl.dataflow.strategy DEBUG: Executor multi has 53 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 53 parsl.dataflow.strategy DEBUG: Executor multi has 53 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76c05d7a05936384ba7c763a199f840c with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a0f16f22-aa3c-4940-b100-45efca7d99f5_calibrate_1123957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a0f16f22-aa3c-4940-b100-45efca7d99f5_calibrate_1123957_26.stderr parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4786a9531cb792ba08b82b32b95e89a0 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fe53a576-5832-4a68-9dda-0997f78d5787_calibrate_1123955_8.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/fe53a576-5832-4a68-9dda-0997f78d5787_calibrate_1123955_8.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03f27093bcd0536b9f36f170ccfd0e02 with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f0ef063f-ab12-494e-a453-be8a7565398e_calibrate_1123953_25.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f0ef063f-ab12-494e-a453-be8a7565398e_calibrate_1123953_25.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd116fe7dec7d45609bc74a17bd3dde8 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/56df859a-4693-4d1c-8467-fc6f23d4ad0f_calibrate_1123955_24.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/56df859a-4693-4d1c-8467-fc6f23d4ad0f_calibrate_1123955_24.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d294b8a6c4e574f0702e43a63ed0ad2d with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3569c69d-9bda-4c80-91ef-57f3102a8e4e_calibrate_1123953_17.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3569c69d-9bda-4c80-91ef-57f3102a8e4e_calibrate_1123953_17.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee7eb00d9b433d56bf9a32acd260d309 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/31e37cf7-9dfb-4b66-b654-af4785298a97_calibrate_1123957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/31e37cf7-9dfb-4b66-b654-af4785298a97_calibrate_1123957_9.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 462984884b78e060d3ec6e2b0694f13c with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/983c83c5-b445-468b-a02f-e6356a26fd92_calibrate_1123953_39.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/983c83c5-b445-468b-a02f-e6356a26fd92_calibrate_1123953_39.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d27715cfe1d3fa68803a4564e55366 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a12b20e1-2aa1-40af-acfc-b820674d02a7_calibrate_1123955_50.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/a12b20e1-2aa1-40af-acfc-b820674d02a7_calibrate_1123955_50.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 681ea50919690615eea07d093f6a05e1 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5171c05c-354d-458d-8f27-37ae3c6fffd7_calibrate_1123957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/5171c05c-354d-458d-8f27-37ae3c6fffd7_calibrate_1123957_46.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b0898b48d7aa447b24f1f3f462e165c with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4acd0671-f5e8-4161-9ace-ba9c76ec89aa_calibrate_1123957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4acd0671-f5e8-4161-9ace-ba9c76ec89aa_calibrate_1123957_23.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11eff73aaf22d79bae89138a0a130d34 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f6ddd5d8-2e69-4973-8846-fd5d9300ffee_calibrate_1123953_22.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/f6ddd5d8-2e69-4973-8846-fd5d9300ffee_calibrate_1123953_22.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1c3a70b9930dd2b20b786217d0f0c05 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3f43b769-5147-48b3-b7c1-8419707d0df7_calibrate_1123955_38.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/3f43b769-5147-48b3-b7c1-8419707d0df7_calibrate_1123955_38.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 41 parsl.dataflow.strategy DEBUG: Executor multi has 41 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52bd24b34278da0a0e4e1b7499f98c9a with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/c768863b-504e-4a10-8b6b-205f96dae55c_calibrate_1123953_60.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/c768863b-504e-4a10-8b6b-205f96dae55c_calibrate_1123953_60.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b2c354ca7063e049d2f36203ad2badb with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc01727e-f5dd-4698-aaa6-7c5911a48442_calibrate_1123953_50.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/bc01727e-f5dd-4698-aaa6-7c5911a48442_calibrate_1123953_50.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4f77adb25e6d50a676998bdc573f297 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b1b477a9-02e0-4579-a7da-d1df5b9dfb44_calibrate_1123957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/b1b477a9-02e0-4579-a7da-d1df5b9dfb44_calibrate_1123957_39.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 024dd4cf44f90b87239f0c14c1eff417 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7d86ce83-282f-40e0-b6be-a80ac0f3ad56_calibrate_1123957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/7d86ce83-282f-40e0-b6be-a80ac0f3ad56_calibrate_1123957_38.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a11bd92b2874a0a7c6fe867c741589da with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/db60417f-cdd0-468c-805d-7ee3229735cc_calibrate_1123957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/db60417f-cdd0-468c-805d-7ee3229735cc_calibrate_1123957_17.stderr parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acf129a380bae19f20cdbb373b101172 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7c37c4f1-6fd8-4b33-90b4-1c953d22c6e2_calibrate_1123953_59.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/7c37c4f1-6fd8-4b33-90b4-1c953d22c6e2_calibrate_1123953_59.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6425ba8f181cb392511c8dd66111a2e with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/58167e5f-1615-4eb8-8f8d-9a616b83d15c_calibrate_1123953_47.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/58167e5f-1615-4eb8-8f8d-9a616b83d15c_calibrate_1123953_47.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3e49d4b8360b79fe74089677db1db38 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3e98ecb6-9021-4ece-b900-2f7ddf05a90a_calibrate_1123953_56.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3e98ecb6-9021-4ece-b900-2f7ddf05a90a_calibrate_1123953_56.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 33 parsl.dataflow.strategy DEBUG: Executor multi has 33 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1e6028d4bf48d3f42fc27fc76bbfce6 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/ab126c75-d6e6-4cc1-9b44-d1356c3e483b_calibrate_1123953_58.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/ab126c75-d6e6-4cc1-9b44-d1356c3e483b_calibrate_1123953_58.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48d57ed48ef2ec4570127c258d2817ac with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/76f99030-2b1e-438e-8cf1-8e2c9fd5084d_calibrate_1123955_49.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/76f99030-2b1e-438e-8cf1-8e2c9fd5084d_calibrate_1123955_49.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bab6aa62f9596265d05a7b92ff006323 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ffbfe150-79e5-4cf4-ab92-721fb7d52410_calibrate_1123955_3.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/ffbfe150-79e5-4cf4-ab92-721fb7d52410_calibrate_1123955_3.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90cf519d8af55f4b2ef304a0857a8c59 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/251f99ec-4151-4c08-af1c-0d9f673af503_calibrate_1123953_55.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/251f99ec-4151-4c08-af1c-0d9f673af503_calibrate_1123953_55.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd97fd4f396baa52e83a800640b03c20 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9420b5cd-282b-4896-b803-3504ea94aca0_calibrate_1123957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/9420b5cd-282b-4896-b803-3504ea94aca0_calibrate_1123957_21.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ff588c47f9d5586168e5c3ba5d2085 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/76876c0a-a7e0-48a4-a4fb-6b71f08021c5_calibrate_1123953_12.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/76876c0a-a7e0-48a4-a4fb-6b71f08021c5_calibrate_1123953_12.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9458640c6bdc4d1b6692fc5d445a9b9 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4fd29945-f42e-4820-8aef-41f383cece13_calibrate_1123957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/4fd29945-f42e-4820-8aef-41f383cece13_calibrate_1123957_12.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 26 parsl.dataflow.strategy DEBUG: Executor multi has 26 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f77b5b0332781ed8ff845634e49deb65 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/121d81c3-af55-485b-b6a8-14499c5cc919_calibrate_1123955_46.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/121d81c3-af55-485b-b6a8-14499c5cc919_calibrate_1123955_46.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54226bfdaf5c6819fb56b490286d3b81 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/292c0022-cfc0-4169-a35a-5074984f5635_calibrate_1123955_7.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/292c0022-cfc0-4169-a35a-5074984f5635_calibrate_1123955_7.stderr 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cf4f066afd1bbea4dbbc6ea20f7c29f with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a2f52dc6-7cbe-4450-8887-6473e1f86721_calibrate_1123953_49.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a2f52dc6-7cbe-4450-8887-6473e1f86721_calibrate_1123953_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 301f802eaa605d22db2c0d1b4b1d6d88 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/48683a0c-07ad-445b-87c4-fa67dc3404c1_calibrate_1123955_34.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/48683a0c-07ad-445b-87c4-fa67dc3404c1_calibrate_1123955_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:12 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:13 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 140 try 0 failed parsl.dataflow.dflow INFO: Task 140 marked for retry parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 9e29e4559b2d94513f8520e1039fd502 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6393eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 1 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:13 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:13 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac9ba16794d3c3065a5091bbb597c17a with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/09b8f5bc-816a-416d-a57b-0e422437b708_calibrate_1123955_58.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/09b8f5bc-816a-416d-a57b-0e422437b708_calibrate_1123955_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:13 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:13 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:13 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:13 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:14 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:15 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c09dca698effbf89db7ba0b4d78d7035 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/434e396d-50cb-4859-a61c-e837d41e0c40_calibrate_1123955_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:16 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:17 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39f461a91777e4a9e73eebfcc835ee5b with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/181dbba6-c2c0-4018-8cd5-698e7d857a49_calibrate_1123957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/181dbba6-c2c0-4018-8cd5-698e7d857a49_calibrate_1123957_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:17 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:17 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:17 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:17 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 110da3a3ae3d9fc14dbb7e0f0a8b0ab6 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 70c7f82134bff57bb2ed9bfc0da98dd2 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b628989a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2679a989-ba9f-4532-85f2-a0d4583919c7_calibrate_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2679a989-ba9f-4532-85f2-a0d4583919c7_calibrate_1123955_26.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/595ea87f-3b62-4c53-846f-9a046b2638c0_characterizeImage_1123955_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 266 try 1 failed parsl.dataflow.dflow ERROR: Task 266 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9bcc33a4f38ca75545a81815580759b9 with result from task 266 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/4cc58051-7542-43a1-ac5b-ef8a730c7414_calibrate_1123953_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:19 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2380e60c745ae607acf1ba38f5be5cdb with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 8100d99b92fceaba97490e531ca12cb5 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a8fdf2d3-5f73-44cb-a18b-363a0a04a837_calibrate_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a8fdf2d3-5f73-44cb-a18b-363a0a04a837_calibrate_1123953_5.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/d681275f-be67-4ed1-a653-e15b5fc1039a_characterizeImage_1123953_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:19 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cff5844fcbc3c1e3f8f4aa2308050baf with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/df6c2d8d-168f-455e-83e4-97a2625505c0_calibrate_1123955_54.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/df6c2d8d-168f-455e-83e4-97a2625505c0_calibrate_1123955_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:20 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:20 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:20 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:21 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:22 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:22 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:22 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:22 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:22 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 17 parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 62/0 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 17 parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 62/0 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54b4fdbd1fe314ca6b36a95a66708743 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 50ff5b43bde1e36d51db8f7ecd0c069d parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a5bccaa-853d-4c02-b321-fde76f57eee7_calibrate_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a5bccaa-853d-4c02-b321-fde76f57eee7_calibrate_1123953_18.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/73819588-6826-4631-82db-a3d66971d6e2_characterizeImage_1123953_18.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb52037a0a4744cd8457213b0a84d2e8 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/bfb58b30-3fb0-4aed-87c2-89ba5850b7b4_calibrate_1123955_42.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed1e24022251cfbcf51f147c187a4cd with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 1ddef909d66d5e1ba9e2647855eacddc parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/501c82ab-e4d3-4eb4-ab6c-0fbc1fc33d81_calibrate_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/501c82ab-e4d3-4eb4-ab6c-0fbc1fc33d81_calibrate_1123953_23.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/eeb4e83a-0559-4b44-8d55-d1c71e499238_characterizeImage_1123953_23.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 16 parsl.dataflow.strategy DEBUG: Executor multi has 16 active tasks, 62/0 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c20e1eead23c0304b3b5a9fdecf68b2 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash f0fff6c50833739465019e77c47f92c0 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62809ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/679d478b-7916-4b6e-be0c-95fe331269bd_calibrate_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/679d478b-7916-4b6e-be0c-95fe331269bd_calibrate_1123957_59.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/21307918-e7ba-4aab-96f6-2a11ec9800f3_characterizeImage_1123957_59.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 16 parsl.dataflow.strategy DEBUG: Executor multi has 16 active tasks, 62/0 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3791ad784fa82a0f48917c19df80976 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 2bdf069f6fb0c76f33f8aa85401ac5b6 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/81841c11-d3d2-4568-8b16-6b43ff1dc80f_calibrate_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/81841c11-d3d2-4568-8b16-6b43ff1dc80f_calibrate_1123955_33.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/b7b9f08b-ccbe-4032-897d-37694723c8fe_characterizeImage_1123955_33.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d2b40c17f5ab26685b9ad1441a8518b with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 77c9d6565a7a6ac2cfaf6d9e0719fde2 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639da660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a8e9b902-67ad-4794-8d9b-0a62f27846d2_calibrate_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a8e9b902-67ad-4794-8d9b-0a62f27846d2_calibrate_1123957_28.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/6ffb42a2-f89b-4546-a37c-7ac7382926f1_characterizeImage_1123957_28.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 16 parsl.dataflow.strategy DEBUG: Executor multi has 16 active tasks, 62/0 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897c6ed585c1c2391314d14e8eda8835 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/69f0324c-e28b-4b60-9629-c72f803ac8a1_calibrate_1123953_20.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/69f0324c-e28b-4b60-9629-c72f803ac8a1_calibrate_1123953_20.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6474b7c372920189ee79de61d33b7648 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 65b0f2f530399856ca562c6654ffee22 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b626e0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6972b65e-308e-4e95-ab90-8dec88f8ced1_calibrate_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6972b65e-308e-4e95-ab90-8dec88f8ced1_calibrate_1123955_56.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/5f57104e-a5c7-4b31-bc90-8d26ed66e921_characterizeImage_1123955_56.stderr 2024-03-07 00:46:42 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:46:42 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:43 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:44 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:45 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:45 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:45 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:45 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:45 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:45 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:45 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:46 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1857f6e1e5a64c266aa2be81cf1c40eb with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash cb6523b2684e1d48e94a276cf6b58d4f parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3f0f0cf7-37d7-4821-a6e0-1e28081d63e3_calibrate_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3f0f0cf7-37d7-4821-a6e0-1e28081d63e3_calibrate_1123957_44.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/452c7335-7da2-4eeb-a6fd-2f55efa657ca_characterizeImage_1123957_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:47 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:48 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23c069ba06489649241b3161ff4213b1 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash f85dfa1742ee73653eaef52d82832351 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b63989c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/f2726e7a-6b05-4368-83ee-de5e705651fd_calibrate_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/f2726e7a-6b05-4368-83ee-de5e705651fd_calibrate_1123955_55.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/d90a6cb5-d39c-4b8b-9795-d0a29d98e20c_characterizeImage_1123955_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:49 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:50 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:50 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:50 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:50 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:46:50 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 15 parsl.dataflow.strategy DEBUG: Executor multi has 15 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 15 parsl.dataflow.strategy DEBUG: Executor multi has 15 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 15 parsl.dataflow.strategy DEBUG: Executor multi has 15 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f798b034d9f2b31e11ae711acfa73dcd with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash fac69ae1a73b7ebd1ab1900d635b340d parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/cb6825a9-0d02-44af-8018-f96e48a4da43_calibrate_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/cb6825a9-0d02-44af-8018-f96e48a4da43_calibrate_1123953_43.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123953/b35f6724-14a3-4710-82bc-ef07ca3447fb_characterizeImage_1123953_43.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 15 parsl.dataflow.strategy DEBUG: Executor multi has 15 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46005d271901f66057883f74f687121f with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 36ce3e8ed250f93832ed8676ba4cb2d2 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b6289b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/8575f53d-0b16-43cd-b213-970833dddbfb_calibrate_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/8575f53d-0b16-43cd-b213-970833dddbfb_calibrate_1123955_4.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/ef3430d5-a355-4191-b224-9a20b0ba52e4_characterizeImage_1123955_4.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 15 parsl.dataflow.strategy DEBUG: Executor multi has 15 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1f3810d6e2c064aa39e39a410e7fe46 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 4b800cc83c431f69f0938dc3b619ab29 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b639afa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/979df976-0370-48fc-b3ef-77e314dcc24d_calibrate_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/979df976-0370-48fc-b3ef-77e314dcc24d_calibrate_1123957_24.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123957/fa739eb9-4015-4b12-8cf9-e19ea82cd74f_characterizeImage_1123957_24.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 140 try 1 failed parsl.dataflow.dflow ERROR: Task 140 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9e29e4559b2d94513f8520e1039fd502 with result from task 140 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/0e8f1768-3590-432a-b502-8a1b3f47164c_calibrate_1123957_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 14 parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8100d99b92fceaba97490e531ca12cb5 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a8fdf2d3-5f73-44cb-a18b-363a0a04a837_calibrate_1123953_5.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/a8fdf2d3-5f73-44cb-a18b-363a0a04a837_calibrate_1123953_5.stderr 2024-03-07 00:47:15 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:47:15 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ddef909d66d5e1ba9e2647855eacddc with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/501c82ab-e4d3-4eb4-ab6c-0fbc1fc33d81_calibrate_1123953_23.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/501c82ab-e4d3-4eb4-ab6c-0fbc1fc33d81_calibrate_1123953_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:16 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:17 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:17 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:17 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:17 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:17 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:17 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:17 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50ff5b43bde1e36d51db8f7ecd0c069d with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a5bccaa-853d-4c02-b321-fde76f57eee7_calibrate_1123953_18.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/3a5bccaa-853d-4c02-b321-fde76f57eee7_calibrate_1123953_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:18 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:19 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:20 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:21 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:21 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:21 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:21 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:21 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:21 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:21 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:22 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:23 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:23 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:23 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:23 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:23 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:23 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70c7f82134bff57bb2ed9bfc0da98dd2 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2679a989-ba9f-4532-85f2-a0d4583919c7_calibrate_1123955_26.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/2679a989-ba9f-4532-85f2-a0d4583919c7_calibrate_1123955_26.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 10 parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c9d6565a7a6ac2cfaf6d9e0719fde2 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a8e9b902-67ad-4794-8d9b-0a62f27846d2_calibrate_1123957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/a8e9b902-67ad-4794-8d9b-0a62f27846d2_calibrate_1123957_28.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0fff6c50833739465019e77c47f92c0 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/679d478b-7916-4b6e-be0c-95fe331269bd_calibrate_1123957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/679d478b-7916-4b6e-be0c-95fe331269bd_calibrate_1123957_59.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bdf069f6fb0c76f33f8aa85401ac5b6 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/81841c11-d3d2-4568-8b16-6b43ff1dc80f_calibrate_1123955_33.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/81841c11-d3d2-4568-8b16-6b43ff1dc80f_calibrate_1123955_33.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65b0f2f530399856ca562c6654ffee22 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6972b65e-308e-4e95-ab90-8dec88f8ced1_calibrate_1123955_56.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/6972b65e-308e-4e95-ab90-8dec88f8ced1_calibrate_1123955_56.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb6523b2684e1d48e94a276cf6b58d4f with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3f0f0cf7-37d7-4821-a6e0-1e28081d63e3_calibrate_1123957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/3f0f0cf7-37d7-4821-a6e0-1e28081d63e3_calibrate_1123957_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f85dfa1742ee73653eaef52d82832351 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/f2726e7a-6b05-4368-83ee-de5e705651fd_calibrate_1123955_55.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/f2726e7a-6b05-4368-83ee-de5e705651fd_calibrate_1123955_55.stderr 2024-03-07 00:47:48 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:47:48 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:49 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:50 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fac69ae1a73b7ebd1ab1900d635b340d with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/cb6825a9-0d02-44af-8018-f96e48a4da43_calibrate_1123953_43.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123953/cb6825a9-0d02-44af-8018-f96e48a4da43_calibrate_1123953_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:51 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:51 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:51 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:51 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:51 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:51 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:51 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:52 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:53 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:54 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:54 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:54 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:54 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:54 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:54 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:54 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:55 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:56 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:56 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:56 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:56 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:56 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:47:56 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36ce3e8ed250f93832ed8676ba4cb2d2 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/8575f53d-0b16-43cd-b213-970833dddbfb_calibrate_1123955_4.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/8575f53d-0b16-43cd-b213-970833dddbfb_calibrate_1123955_4.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b800cc83c431f69f0938dc3b619ab29 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/979df976-0370-48fc-b3ef-77e314dcc24d_calibrate_1123957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123957/979df976-0370-48fc-b3ef-77e314dcc24d_calibrate_1123957_24.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 2024-03-07 00:48:21 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:48:21 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:22 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:23 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:24 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:25 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:25 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:25 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:25 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:25 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:25 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:25 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:26 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:27 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:28 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:29 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:29 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:29 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:29 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:29 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d83904d48596e4f1ce4e3c62f762dac with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash ab62e4688deb3a43921a32b83e7509e9 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e8c35d5a-89f4-4677-b353-6d37bb881326_characterizeImage_1123955_61.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/characterizeImage/1123955/e8c35d5a-89f4-4677-b353-6d37bb881326_characterizeImage_1123955_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 2024-03-07 00:48:54 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:48:54 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:54 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:55 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:56 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:56 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:56 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:56 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:56 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:56 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:56 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:57 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:58 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:48:59 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:00 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:01 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:02 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:02 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:02 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:02 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 2024-03-07 00:49:27 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:49:27 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:27 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:27 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:28 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:29 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:29 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:29 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:29 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:29 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:29 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:29 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:30 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:31 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:32 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:32 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:32 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:32 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:32 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 728 try 0 failed parsl.dataflow.dflow INFO: Task 728 marked for retry parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash ab62e4688deb3a43921a32b83e7509e9 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x152b62775260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 1 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:32 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:32 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:33 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:34 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:35 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:35 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:35 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:49:35 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 2024-03-07 00:50:00 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:50:00 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:00 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:00 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:01 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:01 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:01 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:01 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:01 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:01 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:01 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:02 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:02 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:02 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:02 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:02 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:02 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:02 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:03 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882922,16882923,16882924,16882925,16882926,16882927,16882928,16882929,16882931,16882932,16882933,16882934,16882935,16882936,16882937,16882938,16882939,16882940,16882941,16882942,16882943,16882944,16882945,16882946,16882947,16882948,16882949,16882950,16882951,16882952' parsl.providers.slurm.slurm DEBUG: squeue returned 16882946 R 16882947 R 16882948 R 16882949 R 16882950 R 16882951 R 16882952 R 16882936 R 16882937 R 16882938 R 16882939 R 16882940 R 16882941 R 16882942 R 16882943 R 16882944 R 16882945 R 16882925 R 16882926 R 16882927 R 16882928 R 16882929 R 16882931 R 16882932 R 16882933 R 16882934 R 16882935 R 16882922 R 16882923 R 16882924 R parsl.providers.slurm.slurm DEBUG: Updating job 16882946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882928 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882929 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882933 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882924 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:04 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882953,16882954,16882955,16882956,16882957' parsl.providers.slurm.slurm DEBUG: squeue returned 16882953 R 16882954 R 16882955 R 16882956 R 16882957 R parsl.providers.slurm.slurm DEBUG: Updating job 16882953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882957 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:05 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:06 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:06 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:06 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:06 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:06 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:06 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:06 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882958,16882959,16882960,16882961,16882962,16882963,16882964,16882965,16882966,16882967,16882968,16882969,16882970,16882971,16882972,16882973,16882974,16882975,16882976,16882977' parsl.providers.slurm.slurm DEBUG: squeue returned 16882968 R 16882969 R 16882970 R 16882971 R 16882972 R 16882973 R 16882974 R 16882975 R 16882976 R 16882977 R 16882958 R 16882959 R 16882960 R 16882961 R 16882962 R 16882963 R 16882964 R 16882965 R 16882966 R 16882967 R parsl.providers.slurm.slurm DEBUG: Updating job 16882968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882967 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:07 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:08 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:08 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:08 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:08 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING 2024-03-07 00:50:08 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16882978,16882979,16882980,16882981,16882982,16882983' parsl.providers.slurm.slurm DEBUG: squeue returned 16882982 R 16882983 R 16882978 R 16882979 R 16882980 R 16882981 R parsl.providers.slurm.slurm DEBUG: Updating job 16882982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16882981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23276380755472 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 728 try 1 failed parsl.dataflow.dflow ERROR: Task 728 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ab62e4688deb3a43921a32b83e7509e9 with result from task 728 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/submit/DEEP/20220821/B1d/science#step1/20240307T003716Z/logs/calibrate/1123955/881ce0be-6b41-4732-8058-b97012dfa68c_calibrate_1123955_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 608 parsl.dataflow.dflow INFO: Tasks in state States.failed: 71 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 65 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=62, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 62 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 36aa0140752a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 36aa0140752a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09039088e1cc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09039088e1cc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60fc0130e700 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60fc0130e700 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b0933cbe48ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b0933cbe48ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b4fa411876a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b4fa411876a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ceecb3c7fac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ceecb3c7fac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61029f7c9bb1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61029f7c9bb1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff5facbda80d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff5facbda80d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbfa388dc66e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbfa388dc66e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8607f08fa72b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8607f08fa72b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d393b402e6e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d393b402e6e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8206ec9cb1fd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8206ec9cb1fd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76901758793e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76901758793e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6e53ea9df5e4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6e53ea9df5e4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d92c4afa9225 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d92c4afa9225 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f058563ce37b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f058563ce37b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af0bd36b9bbf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af0bd36b9bbf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6a32641e335a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6a32641e335a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61786a6f1797 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61786a6f1797 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 098323589061 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 098323589061 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a749154ec370 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a749154ec370 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c4a1b65874ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c4a1b65874ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c9d811d3ff6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c9d811d3ff6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf34b2140f39 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf34b2140f39 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67ee92e3c329 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67ee92e3c329 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e002d563ccba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e002d563ccba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bdc7ca2cc231 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bdc7ca2cc231 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8b597a7db6b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8b597a7db6b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62c52526c72f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62c52526c72f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bdd981237bf4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bdd981237bf4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ea943e983fb1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ea943e983fb1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00d1800ff70d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00d1800ff70d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 45d0014d88ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 45d0014d88ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a8c13e36031e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a8c13e36031e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4936f2a88e11 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4936f2a88e11 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5be7204fd018 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5be7204fd018 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bb24daa16498 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bb24daa16498 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 93ea795e01ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 93ea795e01ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 776c3a4cb56d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 776c3a4cb56d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 996838a1b1ad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 996838a1b1ad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5290af9e37d1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5290af9e37d1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 16304c5ac54a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 16304c5ac54a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33aead7970c8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33aead7970c8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf8543aa722d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf8543aa722d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c2a8a170e17 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c2a8a170e17 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6fe25d9388ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6fe25d9388ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 81692a772df3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 81692a772df3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da39e154046f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da39e154046f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f38fd8607061 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f38fd8607061 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2a12f1e0100 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2a12f1e0100 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc183f5db967 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc183f5db967 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 25be7cdcabaa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 25be7cdcabaa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69f29f1ce469 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69f29f1ce469 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2861b57be54c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2861b57be54c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 26185f6b6a7a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 26185f6b6a7a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b6c6567791a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b6c6567791a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 97952f381c64 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 97952f381c64 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b91e06f1c70 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b91e06f1c70 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6c6509b910d5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6c6509b910d5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00dfa7c9f007 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00dfa7c9f007 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 42156d89e430 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 42156d89e430 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff481f60ba90 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff481f60ba90 2024-03-07 00:50:33 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 2024-03-07 00:50:33 proc_lsst.multi:201 [INFO] cancelling 16882966 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882966 on provider escience_a40 2024-03-07 00:50:33 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro 2024-03-07 00:50:33 proc_lsst.multi:201 [INFO] cancelling 16882928 on provider astro proc_lsst.multi INFO: cancelling 16882928 on provider astro 2024-03-07 00:50:33 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro 2024-03-07 00:50:33 proc_lsst.multi:201 [INFO] cancelling 16882944 on provider astro proc_lsst.multi INFO: cancelling 16882944 on provider astro 2024-03-07 00:50:33 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 2024-03-07 00:50:33 proc_lsst.multi:201 [INFO] cancelling 16882976 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882976 on provider escience_a40 2024-03-07 00:50:33 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro 2024-03-07 00:50:33 proc_lsst.multi:201 [INFO] cancelling 16882925 on provider astro proc_lsst.multi INFO: cancelling 16882925 on provider astro 2024-03-07 00:50:33 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k 2024-03-07 00:50:33 proc_lsst.multi:201 [INFO] cancelling 16882955 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882955 on provider escience_rtx6k 2024-03-07 00:50:33 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro 2024-03-07 00:50:33 proc_lsst.multi:201 [INFO] cancelling 16882926 on provider astro proc_lsst.multi INFO: cancelling 16882926 on provider astro 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882979 on provider checkpoint proc_lsst.multi INFO: cancelling 16882979 on provider checkpoint 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882923 on provider astro proc_lsst.multi INFO: cancelling 16882923 on provider astro 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882948 on provider astro proc_lsst.multi INFO: cancelling 16882948 on provider astro 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882949 on provider astro proc_lsst.multi INFO: cancelling 16882949 on provider astro 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882973 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882973 on provider escience_a40 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882932 on provider astro proc_lsst.multi INFO: cancelling 16882932 on provider astro 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882954 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882954 on provider escience_rtx6k 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882950 on provider astro proc_lsst.multi INFO: cancelling 16882950 on provider astro 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882981 on provider checkpoint proc_lsst.multi INFO: cancelling 16882981 on provider checkpoint 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882965 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882965 on provider escience_a40 2024-03-07 00:50:34 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint 2024-03-07 00:50:34 proc_lsst.multi:201 [INFO] cancelling 16882980 on provider checkpoint proc_lsst.multi INFO: cancelling 16882980 on provider checkpoint 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882936 on provider astro proc_lsst.multi INFO: cancelling 16882936 on provider astro 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882982 on provider checkpoint proc_lsst.multi INFO: cancelling 16882982 on provider checkpoint 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882942 on provider astro proc_lsst.multi INFO: cancelling 16882942 on provider astro 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882971 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882971 on provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882983 on provider checkpoint proc_lsst.multi INFO: cancelling 16882983 on provider checkpoint 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882938 on provider astro proc_lsst.multi INFO: cancelling 16882938 on provider astro 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882974 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882974 on provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882958 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882958 on provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882941 on provider astro proc_lsst.multi INFO: cancelling 16882941 on provider astro 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882970 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882970 on provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 2024-03-07 00:50:35 proc_lsst.multi:201 [INFO] cancelling 16882977 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882977 on provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882964 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882964 on provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882933 on provider astro proc_lsst.multi INFO: cancelling 16882933 on provider astro 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882956 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882956 on provider escience_rtx6k 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882943 on provider astro proc_lsst.multi INFO: cancelling 16882943 on provider astro 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882940 on provider astro proc_lsst.multi INFO: cancelling 16882940 on provider astro 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882953 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882953 on provider escience_rtx6k 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882947 on provider astro proc_lsst.multi INFO: cancelling 16882947 on provider astro 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882931 on provider astro proc_lsst.multi INFO: cancelling 16882931 on provider astro 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882972 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882972 on provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882961 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882961 on provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 2024-03-07 00:50:36 proc_lsst.multi:201 [INFO] cancelling 16882975 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882975 on provider escience_a40 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882934 on provider astro proc_lsst.multi INFO: cancelling 16882934 on provider astro 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882978 on provider checkpoint proc_lsst.multi INFO: cancelling 16882978 on provider checkpoint 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882963 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882963 on provider escience_a40 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882969 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882969 on provider escience_a40 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882922 on provider astro proc_lsst.multi INFO: cancelling 16882922 on provider astro 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 44080 on provider local proc_lsst.multi INFO: cancelling 44080 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 44080 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882929 on provider astro proc_lsst.multi INFO: cancelling 16882929 on provider astro 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882935 on provider astro proc_lsst.multi INFO: cancelling 16882935 on provider astro 2024-03-07 00:50:37 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 2024-03-07 00:50:37 proc_lsst.multi:201 [INFO] cancelling 16882962 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882962 on provider escience_a40 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882967 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882967 on provider escience_a40 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882927 on provider astro proc_lsst.multi INFO: cancelling 16882927 on provider astro 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882960 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882960 on provider escience_a40 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882959 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882959 on provider escience_a40 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882952 on provider astro proc_lsst.multi INFO: cancelling 16882952 on provider astro 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882957 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882957 on provider escience_rtx6k 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882937 on provider astro proc_lsst.multi INFO: cancelling 16882937 on provider astro 2024-03-07 00:50:38 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro 2024-03-07 00:50:38 proc_lsst.multi:201 [INFO] cancelling 16882924 on provider astro proc_lsst.multi INFO: cancelling 16882924 on provider astro 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882968 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882968 on provider escience_a40 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882939 on provider astro proc_lsst.multi INFO: cancelling 16882939 on provider astro 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882946 on provider astro proc_lsst.multi INFO: cancelling 16882946 on provider astro 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882945 on provider astro proc_lsst.multi INFO: cancelling 16882945 on provider astro 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882951 on provider astro proc_lsst.multi INFO: cancelling 16882951 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 00:50:39 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 00:50:39 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16882978', '16882979', '16882980', '16882981', '16882982', '16882983', '16882922', '16882923', '16882924', '16882925', '16882926', '16882927', '16882928', '16882929', '16882931', '16882932', '16882933', '16882934', '16882935', '16882936', '16882937', '16882938', '16882939', '16882940', '16882941', '16882942', '16882943', '16882944', '16882945', '16882946', '16882947', '16882948', '16882949', '16882950', '16882951', '16882952', '16882958', '16882959', '16882960', '16882961', '16882962', '16882963', '16882964', '16882965', '16882966', '16882967', '16882968', '16882969', '16882970', '16882971', '16882972', '16882973', '16882974', '16882975', '16882976', '16882977', '16882953', '16882954', '16882955', '16882956', '16882957', '44080'] proc_lsst.multi INFO: new jobs since last cancel ['16882978', '16882979', '16882980', '16882981', '16882982', '16882983', '16882922', '16882923', '16882924', '16882925', '16882926', '16882927', '16882928', '16882929', '16882931', '16882932', '16882933', '16882934', '16882935', '16882936', '16882937', '16882938', '16882939', '16882940', '16882941', '16882942', '16882943', '16882944', '16882945', '16882946', '16882947', '16882948', '16882949', '16882950', '16882951', '16882952', '16882958', '16882959', '16882960', '16882961', '16882962', '16882963', '16882964', '16882965', '16882966', '16882967', '16882968', '16882969', '16882970', '16882971', '16882972', '16882973', '16882974', '16882975', '16882976', '16882977', '16882953', '16882954', '16882955', '16882956', '16882957', '44080'] 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882978 in provider checkpoint proc_lsst.multi INFO: found job 16882978 in provider checkpoint 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882978 on provider checkpoint proc_lsst.multi INFO: cancelling 16882978 on provider checkpoint 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882979 in provider checkpoint proc_lsst.multi INFO: found job 16882979 in provider checkpoint 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882979 on provider checkpoint proc_lsst.multi INFO: cancelling 16882979 on provider checkpoint 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882980 in provider checkpoint proc_lsst.multi INFO: found job 16882980 in provider checkpoint 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882980 on provider checkpoint proc_lsst.multi INFO: cancelling 16882980 on provider checkpoint 2024-03-07 00:50:39 proc_lsst.multi:146 [INFO] found job 16882981 in provider checkpoint proc_lsst.multi INFO: found job 16882981 in provider checkpoint 2024-03-07 00:50:39 proc_lsst.multi:201 [INFO] cancelling 16882981 on provider checkpoint proc_lsst.multi INFO: cancelling 16882981 on provider checkpoint 2024-03-07 00:50:40 proc_lsst.multi:146 [INFO] found job 16882982 in provider checkpoint proc_lsst.multi INFO: found job 16882982 in provider checkpoint 2024-03-07 00:50:40 proc_lsst.multi:201 [INFO] cancelling 16882982 on provider checkpoint proc_lsst.multi INFO: cancelling 16882982 on provider checkpoint 2024-03-07 00:50:40 proc_lsst.multi:146 [INFO] found job 16882983 in provider checkpoint proc_lsst.multi INFO: found job 16882983 in provider checkpoint 2024-03-07 00:50:40 proc_lsst.multi:201 [INFO] cancelling 16882983 on provider checkpoint proc_lsst.multi INFO: cancelling 16882983 on provider checkpoint 2024-03-07 00:50:40 proc_lsst.multi:146 [INFO] found job 16882922 in provider astro proc_lsst.multi INFO: found job 16882922 in provider astro 2024-03-07 00:50:40 proc_lsst.multi:201 [INFO] cancelling 16882922 on provider astro proc_lsst.multi INFO: cancelling 16882922 on provider astro 2024-03-07 00:50:40 proc_lsst.multi:146 [INFO] found job 16882923 in provider astro proc_lsst.multi INFO: found job 16882923 in provider astro 2024-03-07 00:50:40 proc_lsst.multi:201 [INFO] cancelling 16882923 on provider astro proc_lsst.multi INFO: cancelling 16882923 on provider astro 2024-03-07 00:50:40 proc_lsst.multi:146 [INFO] found job 16882924 in provider astro proc_lsst.multi INFO: found job 16882924 in provider astro 2024-03-07 00:50:40 proc_lsst.multi:201 [INFO] cancelling 16882924 on provider astro proc_lsst.multi INFO: cancelling 16882924 on provider astro 2024-03-07 00:50:40 proc_lsst.multi:146 [INFO] found job 16882925 in provider astro proc_lsst.multi INFO: found job 16882925 in provider astro 2024-03-07 00:50:40 proc_lsst.multi:201 [INFO] cancelling 16882925 on provider astro proc_lsst.multi INFO: cancelling 16882925 on provider astro 2024-03-07 00:50:40 proc_lsst.multi:146 [INFO] found job 16882926 in provider astro proc_lsst.multi INFO: found job 16882926 in provider astro 2024-03-07 00:50:40 proc_lsst.multi:201 [INFO] cancelling 16882926 on provider astro proc_lsst.multi INFO: cancelling 16882926 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882927 in provider astro proc_lsst.multi INFO: found job 16882927 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882927 on provider astro proc_lsst.multi INFO: cancelling 16882927 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882928 in provider astro proc_lsst.multi INFO: found job 16882928 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882928 on provider astro proc_lsst.multi INFO: cancelling 16882928 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882929 in provider astro proc_lsst.multi INFO: found job 16882929 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882929 on provider astro proc_lsst.multi INFO: cancelling 16882929 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882931 in provider astro proc_lsst.multi INFO: found job 16882931 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882931 on provider astro proc_lsst.multi INFO: cancelling 16882931 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882932 in provider astro proc_lsst.multi INFO: found job 16882932 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882932 on provider astro proc_lsst.multi INFO: cancelling 16882932 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882933 in provider astro proc_lsst.multi INFO: found job 16882933 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882933 on provider astro proc_lsst.multi INFO: cancelling 16882933 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882934 in provider astro proc_lsst.multi INFO: found job 16882934 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882934 on provider astro proc_lsst.multi INFO: cancelling 16882934 on provider astro 2024-03-07 00:50:41 proc_lsst.multi:146 [INFO] found job 16882935 in provider astro proc_lsst.multi INFO: found job 16882935 in provider astro 2024-03-07 00:50:41 proc_lsst.multi:201 [INFO] cancelling 16882935 on provider astro proc_lsst.multi INFO: cancelling 16882935 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882936 in provider astro proc_lsst.multi INFO: found job 16882936 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882936 on provider astro proc_lsst.multi INFO: cancelling 16882936 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882937 in provider astro proc_lsst.multi INFO: found job 16882937 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882937 on provider astro proc_lsst.multi INFO: cancelling 16882937 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882938 in provider astro proc_lsst.multi INFO: found job 16882938 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882938 on provider astro proc_lsst.multi INFO: cancelling 16882938 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882939 in provider astro proc_lsst.multi INFO: found job 16882939 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882939 on provider astro proc_lsst.multi INFO: cancelling 16882939 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882940 in provider astro proc_lsst.multi INFO: found job 16882940 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882940 on provider astro proc_lsst.multi INFO: cancelling 16882940 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882941 in provider astro proc_lsst.multi INFO: found job 16882941 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882941 on provider astro proc_lsst.multi INFO: cancelling 16882941 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882942 in provider astro proc_lsst.multi INFO: found job 16882942 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882942 on provider astro proc_lsst.multi INFO: cancelling 16882942 on provider astro 2024-03-07 00:50:42 proc_lsst.multi:146 [INFO] found job 16882943 in provider astro proc_lsst.multi INFO: found job 16882943 in provider astro 2024-03-07 00:50:42 proc_lsst.multi:201 [INFO] cancelling 16882943 on provider astro proc_lsst.multi INFO: cancelling 16882943 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882944 in provider astro proc_lsst.multi INFO: found job 16882944 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882944 on provider astro proc_lsst.multi INFO: cancelling 16882944 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882945 in provider astro proc_lsst.multi INFO: found job 16882945 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882945 on provider astro proc_lsst.multi INFO: cancelling 16882945 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882946 in provider astro proc_lsst.multi INFO: found job 16882946 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882946 on provider astro proc_lsst.multi INFO: cancelling 16882946 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882947 in provider astro proc_lsst.multi INFO: found job 16882947 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882947 on provider astro proc_lsst.multi INFO: cancelling 16882947 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882948 in provider astro proc_lsst.multi INFO: found job 16882948 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882948 on provider astro proc_lsst.multi INFO: cancelling 16882948 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882949 in provider astro proc_lsst.multi INFO: found job 16882949 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882949 on provider astro proc_lsst.multi INFO: cancelling 16882949 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882950 in provider astro proc_lsst.multi INFO: found job 16882950 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882950 on provider astro proc_lsst.multi INFO: cancelling 16882950 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882951 in provider astro proc_lsst.multi INFO: found job 16882951 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882951 on provider astro proc_lsst.multi INFO: cancelling 16882951 on provider astro 2024-03-07 00:50:43 proc_lsst.multi:146 [INFO] found job 16882952 in provider astro proc_lsst.multi INFO: found job 16882952 in provider astro 2024-03-07 00:50:43 proc_lsst.multi:201 [INFO] cancelling 16882952 on provider astro proc_lsst.multi INFO: cancelling 16882952 on provider astro 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882958 in provider escience_a40 proc_lsst.multi INFO: found job 16882958 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882958 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882958 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882959 in provider escience_a40 proc_lsst.multi INFO: found job 16882959 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882959 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882959 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882960 in provider escience_a40 proc_lsst.multi INFO: found job 16882960 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882960 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882960 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882961 in provider escience_a40 proc_lsst.multi INFO: found job 16882961 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882961 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882961 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882962 in provider escience_a40 proc_lsst.multi INFO: found job 16882962 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882962 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882962 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882963 in provider escience_a40 proc_lsst.multi INFO: found job 16882963 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882963 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882963 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882964 in provider escience_a40 proc_lsst.multi INFO: found job 16882964 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882964 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882964 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882965 in provider escience_a40 proc_lsst.multi INFO: found job 16882965 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882965 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882965 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882966 in provider escience_a40 proc_lsst.multi INFO: found job 16882966 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882966 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882966 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882967 in provider escience_a40 proc_lsst.multi INFO: found job 16882967 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882967 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882967 on provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:146 [INFO] found job 16882968 in provider escience_a40 proc_lsst.multi INFO: found job 16882968 in provider escience_a40 2024-03-07 00:50:44 proc_lsst.multi:201 [INFO] cancelling 16882968 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882968 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882969 in provider escience_a40 proc_lsst.multi INFO: found job 16882969 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882969 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882969 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882970 in provider escience_a40 proc_lsst.multi INFO: found job 16882970 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882970 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882970 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882971 in provider escience_a40 proc_lsst.multi INFO: found job 16882971 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882971 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882971 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882972 in provider escience_a40 proc_lsst.multi INFO: found job 16882972 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882972 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882972 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882973 in provider escience_a40 proc_lsst.multi INFO: found job 16882973 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882973 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882973 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882974 in provider escience_a40 proc_lsst.multi INFO: found job 16882974 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882974 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882974 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882975 in provider escience_a40 proc_lsst.multi INFO: found job 16882975 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882975 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882975 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882976 in provider escience_a40 proc_lsst.multi INFO: found job 16882976 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882976 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882976 on provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:146 [INFO] found job 16882977 in provider escience_a40 proc_lsst.multi INFO: found job 16882977 in provider escience_a40 2024-03-07 00:50:45 proc_lsst.multi:201 [INFO] cancelling 16882977 on provider escience_a40 proc_lsst.multi INFO: cancelling 16882977 on provider escience_a40 2024-03-07 00:50:47 proc_lsst.multi:146 [INFO] found job 16882953 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882953 in provider escience_rtx6k 2024-03-07 00:50:47 proc_lsst.multi:201 [INFO] cancelling 16882953 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882953 on provider escience_rtx6k 2024-03-07 00:50:47 proc_lsst.multi:146 [INFO] found job 16882954 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882954 in provider escience_rtx6k 2024-03-07 00:50:47 proc_lsst.multi:201 [INFO] cancelling 16882954 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882954 on provider escience_rtx6k 2024-03-07 00:50:48 proc_lsst.multi:146 [INFO] found job 16882955 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882955 in provider escience_rtx6k 2024-03-07 00:50:48 proc_lsst.multi:201 [INFO] cancelling 16882955 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882955 on provider escience_rtx6k 2024-03-07 00:50:48 proc_lsst.multi:146 [INFO] found job 16882956 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882956 in provider escience_rtx6k 2024-03-07 00:50:48 proc_lsst.multi:201 [INFO] cancelling 16882956 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882956 on provider escience_rtx6k 2024-03-07 00:50:48 proc_lsst.multi:146 [INFO] found job 16882957 in provider escience_rtx6k proc_lsst.multi INFO: found job 16882957 in provider escience_rtx6k 2024-03-07 00:50:48 proc_lsst.multi:201 [INFO] cancelling 16882957 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16882957 on provider escience_rtx6k 2024-03-07 00:50:49 proc_lsst.multi:146 [INFO] found job 44080 in provider local proc_lsst.multi INFO: found job 44080 in provider local 2024-03-07 00:50:49 proc_lsst.multi:201 [INFO] cancelling 44080 on provider local proc_lsst.multi INFO: cancelling 44080 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 44080 parsl.providers.local.local WARNING: Failed to kill PID: 44080 and child processes on local 2024-03-07 00:50:50 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 768.5566 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220821_B1d_science#step1_20240307T003716Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 768.5630 seconds; current memory usage: 0.359 Gibyte, delta: -0.011 Gibyte, peak delta: 0.006 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 804.7989 seconds; current memory usage: 0.359 Gibyte, delta: 0.223 Gibyte, peak delta: 0.239 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.376 Gibyte (main), 0.376 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220821_B1d_science#step1_20240307T003716Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up